Python code snippets

Deferred Plugins

Implementing a deferred extraction plugin requires inheriting the DeferredExtractionPlugin base class.

class DeferredPlugin(DeferredExtractionPlugin):
    def process(self, trace, context, searcher):

This allows accessing a third TraceSearcher parameter in the process function. This can be used to search for traces:

with searcher.search('file.extension:html', 10) as searchresult:
    for trace in searchresult:
        log.debug(f'extension {trace.get("file.extension")}')

The search method accepts two arguments; a HQL query and the maximum number of traces the return. The search method accepts an HQL query and a count, which represents the maximum number of traces to return.

It may be useful to specifically search for traces from the image being extracted. Add "image:" + trace.get("image") to your query. The query of the provided example could be extended like this: "file.extension:html AND image:" + trace.get("image").

The returned SearchResult should be closed, for example by using with. The resulting search result is an iterable, which will be exhausted when no more traces are available. The search result allows taking one or more traces by calling :py: meth:take <hansken_extraction_plugin.api.search_result.SearchResult.take> or takeone.

Adding properties to a trace

Use update to add trace types and their properties to an :py: class:ExtractionTrace <hansken_extraction_plugin.api.extraction_trace.ExtractionTrace>. Example:

def process(self, trace, data_context):
    # get the name of the file
    file_name = trace.get('file.name')
    # set the chat application property on the trace
    trace.update('chatConversation.application', f'DemoApp {file_name}')

Hansken trace model documentation is available for a complete overview of available trace-types and their properties. Use it when updating a trace. Each property has an expected data-type, which is either string, Date, integer or boolean. This property data-type information also can be found in the trace model documentation.

Date properties

When adding a property which holds a value of data-type Date, always define timezone as being UTC. Example:

def process(self, trace, data_context):
    trace.update('file.modifiedOn',
                 datetime.fromtimestamp(1630510809, tz=timezone.utc))

Category for extra properties

If the information, which must be added as a property, does not match any of the existing properties of Hansken trace model, use the category “misc” (miscellaneous). When part of the category “misc”, any name can be given to a property. The values of miscellaneous properties are expected to be of data-type string. Example:

def process(self, trace, data_context):
    trace.update({
        'file.misc.notes': 'Some additional notes about the file trace.',
        'file.misc.anyName': 'Even more notes.'
    })

Adding tracelets

In the following Python example, a “prediction” tracelet is added to a trace. The tracelet consists of a list of four properties, namely “class”, “confidence”, “modelName” and “modelVersion”.

trace.add_tracelet(Tracelet('prediction', {'class': 'telephone',
                                           'confidence': 0.8,
                                           'modelName': 'yolo',
                                           'modelVersion': '2.0'}))

Adding child traces to a trace

Adding child traces to the trace can be done by creating a builder with child_builder. Example:

def process(self, trace, data_context):
    child_builder = trace.child_builder('childTrace-1')
    child_builder.update({
        'chatMessage.application': 'DemoApp',
        'chatMessage.from': 'Ann',
        'chatMessage.to': ['Mark'],
        # list, because there can be multiple receivers
        'chatMessage.message': 'Hello, are you there?',
    }).build()

This adds a single child trace with name childTrace-1 and four properties.

Adding data to a trace

Traces can have data attached to them. See Data streams for more information. The following two snippets demonstrate how to add data to a trace.

It is currently not possible to verify that a specific data stream is already set or not.

Data Transformations

The most efficient way to add data to a trace is using data transformations. See Data Transformations for more details.

The following example sets a new datastream with dataType html on a trace, by setting a ranged data transformation:

trace.add_transformation('html', RangedTransformation(Range(offset, length)))

The following example creates a child trace and sets a new datastream with dataType raw on it, by setting a ranged data transformation with two ranges:

child = trace.child_builder('new trace')
child.add_transformation('raw', RangedTransformation.builder()
                         .add_range(10, 20)
                         .add_range(50, 30)
                         .build())
});

Blobs

It is not always possible to create a transformation for the data that has to be added to a trace. For example, if the data is a result of a computation, and not a direct subset of another data stream..

The following snippet shows how to create a new data stream of dataType raw on a trace from a blob stored in bytes:

data = {'raw': b'...'}
trace.update(data=data);

Logging

We use Logbook to log messages in Python. Logbook is a logging system for Python that replaces the standard library’s logging module.

To enable logging in your plugin, add the following to the top of your plugin code:

from logbook import Logger

log = Logger(__name__)

From there on the logging is pretty straight forward:

log.info(f'Logging a variable: {my_variable}')

The default log level is WARNING. You can use the -v option of serve_plugin.py to increase the log level. This is typically done in the plugin Dockerfile.

Warning

Be careful with logging sensitive information.

Note

A lot of logging examples can be found in the Extraction Plugin Examples.

Note

Contact your Hansken administrator for more information on where to find logs for your Hansken environment.

Specifying system resources

It is possible to specify maximum system resources in the PluginInfo. To run a plugin with 0.5 cpu (= 0.5 vCPU/Core/hyperthread) and 1 gb memory, for example, the following configuration can be added to PluginInfo:

plugin_info = PluginInfo(...,
                         resources=PluginResources(maximum_cpu=0.5, maximum_memory=1000))