0.6.3

0.6.3/_sources/dev/python/api/hansken_extraction_plugin.api.data_context.rst.txt

@@ -0,0 +1,29 @@
+hansken\_extraction\_plugin.api.data\_context
+=============================================
+
+.. automodule:: hansken_extraction_plugin.api.data_context
+
+   
+   
+   
+
+   
+   
+   
+
+   
+   
+   .. rubric:: Classes
+
+   .. autosummary::
+   
+      DataContext
+   
+   
+
+   
+   
+   
+
+
+

0.6.3/_sources/dev/python/api/hansken_extraction_plugin.api.extraction_plugin.rst.txt

@@ -0,0 +1,32 @@
+hansken\_extraction\_plugin.api.extraction\_plugin
+==================================================
+
+.. automodule:: hansken_extraction_plugin.api.extraction_plugin
+
+   
+   
+   
+
+   
+   
+   
+
+   
+   
+   .. rubric:: Classes
+
+   .. autosummary::
+   
+      BaseExtractionPlugin
+      DeferredExtractionPlugin
+      ExtractionPlugin
+      MetaExtractionPlugin
+   
+   
+
+   
+   
+   
+
+
+

0.6.3/_sources/dev/python/api/hansken_extraction_plugin.api.extraction_trace.rst.txt

@@ -0,0 +1,33 @@
+hansken\_extraction\_plugin.api.extraction\_trace
+=================================================
+
+.. automodule:: hansken_extraction_plugin.api.extraction_trace
+
+   
+   
+   
+
+   
+   
+   
+
+   
+   
+   .. rubric:: Classes
+
+   .. autosummary::
+   
+      ExtractionTrace
+      ExtractionTraceBuilder
+      MetaExtractionTrace
+      SearchTrace
+      Trace
+   
+   
+
+   
+   
+   
+
+
+

0.6.3/_sources/dev/python/api/hansken_extraction_plugin.api.plugin_info.rst.txt

@@ -0,0 +1,33 @@
+hansken\_extraction\_plugin.api.plugin\_info
+============================================
+
+.. automodule:: hansken_extraction_plugin.api.plugin_info
+
+   
+   
+   
+
+   
+   
+   
+
+   
+   
+   .. rubric:: Classes
+
+   .. autosummary::
+   
+      Author
+      MaturityLevel
+      PluginId
+      PluginInfo
+      PluginResources
+   
+   
+
+   
+   
+   
+
+
+

0.6.3/_sources/dev/python/api/hansken_extraction_plugin.api.rst.txt

@@ -0,0 +1,38 @@
+hansken\_extraction\_plugin.api
+===============================
+
+.. automodule:: hansken_extraction_plugin.api
+
+   
+   
+   
+
+   
+   
+   
+
+   
+   
+   
+
+   
+   
+   
+
+
+
+.. rubric:: Modules
+
+.. autosummary::
+   :toctree:
+   :recursive:
+
+   hansken_extraction_plugin.api.data_context
+   hansken_extraction_plugin.api.extraction_plugin
+   hansken_extraction_plugin.api.extraction_trace
+   hansken_extraction_plugin.api.plugin_info
+   hansken_extraction_plugin.api.search_result
+   hansken_extraction_plugin.api.trace_searcher
+   hansken_extraction_plugin.api.tracelet
+   hansken_extraction_plugin.api.transformation
+

0.6.3/_sources/dev/python/api/hansken_extraction_plugin.api.search_result.rst.txt

@@ -0,0 +1,29 @@
+hansken\_extraction\_plugin.api.search\_result
+==============================================
+
+.. automodule:: hansken_extraction_plugin.api.search_result
+
+   
+   
+   
+
+   
+   
+   
+
+   
+   
+   .. rubric:: Classes
+
+   .. autosummary::
+   
+      SearchResult
+   
+   
+
+   
+   
+   
+
+
+

0.6.3/_sources/dev/python/api/hansken_extraction_plugin.api.trace_searcher.rst.txt

@@ -0,0 +1,29 @@
+hansken\_extraction\_plugin.api.trace\_searcher
+===============================================
+
+.. automodule:: hansken_extraction_plugin.api.trace_searcher
+
+   
+   
+   
+
+   
+   
+   
+
+   
+   
+   .. rubric:: Classes
+
+   .. autosummary::
+   
+      TraceSearcher
+   
+   
+
+   
+   
+   
+
+
+

0.6.3/_sources/dev/python/api/hansken_extraction_plugin.api.tracelet.rst.txt

@@ -0,0 +1,29 @@
+hansken\_extraction\_plugin.api.tracelet
+========================================
+
+.. automodule:: hansken_extraction_plugin.api.tracelet
+
+   
+   
+   
+
+   
+   
+   
+
+   
+   
+   .. rubric:: Classes
+
+   .. autosummary::
+   
+      Tracelet
+   
+   
+
+   
+   
+   
+
+
+

0.6.3/_sources/dev/python/api/hansken_extraction_plugin.api.transformation.rst.txt

@@ -0,0 +1,31 @@
+hansken\_extraction\_plugin.api.transformation
+==============================================
+
+.. automodule:: hansken_extraction_plugin.api.transformation
+
+   
+   
+   
+
+   
+   
+   
+
+   
+   
+   .. rubric:: Classes
+
+   .. autosummary::
+   
+      Range
+      RangedTransformation
+      Transformation
+   
+   
+
+   
+   
+   
+
+
+

0.6.3/_sources/dev/python/api_changelog.md.txt

@@ -0,0 +1,322 @@
+# Python API Changelog
+
+This document summarizes all important API changes in the Extraction Plugin API. This document only shows changes that
+are important to plugin developers. For a full list of changes per version, please refer to the general
+:ref:`changelog <changelog>`.
+
+.. If present, remove `..` before `## |version|` if you create a new entry after a previous release.
+
+.. ## |version|
+
+
+## 0.6.1
+
+The docker image build script `build_plugin.py` has been updated to allow for extension of the docker command. 
+This can be especially handy for specifying a proxy. You should build your plugin container image with the following 
+command:
+
+```bash
+build_plugin PLUGIN_FILE DOCKER_FILE_DIRECTORY [DOCKER_IMAGE_NAME] [DOCKER_ARGS]
+```
+
+.. warning:: Note that the `DOCKER_IMAGE_NAME` argument no longer requires a `-n` parameter to be specified.
+
+For usage read further in [packaging](packaging.md).
+
+## 0.6.0
+
+.. warning:: This is an API breaking change.
+   Upgrading your plugin to this version will require code changes.
+   Plugins built with previous versions of the SDK from `0.3.0` will still work with Hansken.
+
+.. warning:: It is strongly recommended to upgrade your plugins to this new version because it significantly improves
+   the start-up time of Hansken. See the migration steps below.
+
+This release contains both build pipeline changes and API changes.
+Please read all changes carefully.
+
+### Build pipeline change
+
+* Extraction plugin container images are now labeled with PluginInfo. This
+  allows Hansken to efficiently load extraction plugins.
+  Migration steps from earlier versions:
+
+  1. Update the SDK version in your `setup.py` / `requirements.txt`
+  2. If you come from a version prior to `0.4.0`, or if you use a plugin name
+     instead of a plugin id in your `pluginInfo()`, switch to the plugin id style
+     (read  instructions for version `0.4.0`)
+  3. Update your build scripts to build your plugin (Docker) container image.
+     Be sure to [have the Extraction Plugins SDK installed](getting_started.md#Installation).
+     Then, you should build your plugin container image with the following command:
+
+     ```bash
+     build_plugin PLUGIN_FILE DOCKER_FILE_DIRECTORY -n [DOCKER_IMAGE_NAME]
+     ```
+
+     For example:
+     ```bash
+     build_plugin plugin/chatplugin.py . -n extraction-plugins/chatplugin
+     ```
+
+     This will generate a plugin image:
+
+    * The extraction plugin is added to your local image registry (`docker images`),
+    * Note that DOCKER\_IMAGE\_NAME is optional and will default to `extraction-plugin/PLUGINID`, e.g.
+     `extraction-plugin/nfi.nl/extract/chat/whatsapp`,
+    * The image is tagged with two tags: `latest`, and your plugin version.
+
+
+### API changes
+
+* The field `plugin` has been removed from `PluginInfo`.
+* The field `pluginId` should now be the first argument of PluginInfo (when using unnamed arguments).
+
+  Old (unnamed arguments):
+
+  ```python
+  def plugin_info(self):
+      return PluginInfo(self, '1.0.0', 'description', author,
+                        MaturityLevel.PROOF_OF_CONCEPT, '*, 'https://hansken.org',
+                        PluginId(...), 'Apache License 2.0')
+  ```
+
+  New (removed `self`, and moved `PluginId(...)` to first argument position):
+
+  ```python
+  def plugin_info(self):
+      return PluginInfo(PluginId(...), '1.0.0', 'description',
+                        author, MaturityLevel.PROOF_OF_CONCEPT,
+                        '*', 'https://hansken.org', 'Apache License 2.0')
+  ```
+
+  Old (named arguments):
+
+  ```python
+  def plugin_info(self):
+      return PluginInfo(plugin=self,
+                        version='1.0.0',
+                        ...)
+  ```
+
+  New (removed `plugin=self`):
+
+  ```python
+  def plugin_info(self):
+      return PluginInfo(version='1.0.0',
+                        ...)
+  ```
+
+* Plugin `data_context.data_size` is now a variable instead of a method:
+
+  Old:
+
+  ```python
+  def process(self, trace: ExtractionTrace, data_context: DataContext):
+      size = data_context.data_size()
+  ```
+
+  New:
+
+  ```python
+  def process(self, trace: ExtractionTrace, data_context: DataContext):
+      size = data_context.data_size
+  ```
+
+* Simplify declaring required runtime resources in a plugin's info.
+
+  Extraction plugin resources don't use the builder pattern anymore.
+
+  Old:
+
+  ```python
+  return PluginInfo(
+      ...,
+      resources=PluginResources.builder().maximum_cpu(0.5).maximum_memory(1000).build())
+  )
+  ```
+
+  New:
+
+  ```python
+  # no need for a builder, declare resources by direct instantiation
+  return PluginInfo(
+      ...,
+      resources=PluginResources(maximum_cpu=2.0, maximum_memory=2048)
+  )
+  # or, as before, specify just on resource
+  return PluginInfo(
+      ...,
+      resources=PluginResources(maximum_memory=4096)
+  )
+  ```
+
+## 0.5.1
+
+* Simplify tracelet properties by making the tracelet type prefix optional.
+
+  ```python
+  # using a Tracelet object
+  trace.add_tracelet(Tracelet("prediction", {
+      "type": "example",
+      "confidence": 0.8
+  }))
+  # or without a Tracelet object
+  trace.add_tracelet("identity", {"name": "John Doe", "status": "online"})
+  ```
+
+* Enabled _manual_ plugin testing, as described on :ref:`advanced use of the test framework in Python<python testing>`.
+
+## 0.5.0
+
+* Support vector data type in trace properties.
+
+  ```python
+  embedding = Vector.from_sequence((width, height))
+  tracelet = Tracelet("prediction", {
+      "prediction.type": "example-vector",
+      "prediction.embedding": embedding
+  })
+  trace.add_tracelet(tracelet)
+  ```
+
+## 0.4.13
+
+* When writing input search traces for tests, it is no longer required to explicitly set an `id` property.
+  These are automatically generated when executing tests.
+
+## 0.4.7
+
+* More `$data` matchers are supported in Hansken.py plugin runner. Before this improvement it was only possible to match
+  on `$data.type`. Now it is also possible to match for example on `$data.mimeType` and `$data.mimeClass`. The `$data`
+  matcher should still be at the end of the query as before.
+
+## 0.4.6
+
+* It is now possible to specify maximum system resources in the `PluginInfo`. To run a plugin with 0.5 cpu (= 0.5
+  vCPU/Core/hyperthread) and 1 gb memory, for example, the following configuration can be added to `PluginInfo`:
+
+  ```python
+  plugin_info = PluginInfo(...,
+                           resources=PluginResources.builder().maximum_cpu(0.5).maximum_memory(1000).build())
+  ```
+
+## 0.4.0
+
+* Extraction Plugins are now identified with a `PluginInfo.PluginId` containing a domain, category and name. The
+  method `PluginInfo.name(pluginName)` has been replaced by `PluginInfo.id(new PluginId(domain, category, name)`. More
+  details on the plugin naming conventions can be found at the :doc:`../concepts/plugin_naming_convention` section.
+
+* `PluginInfo.name()` is now deprecated (but will still work for backwards compatibility).
+
+* A new license field `PluginInfo.license` has also been added in this release.
+
+* The following example creates a PluginInfo for a plugin with the name `TestPlugin`, licensed under
+  the `Apache License 2.0` license:
+
+  ```python
+  class TestPlugin(ExtractionPlugin):
+      def plugin_info(self) -> PluginInfo:
+          return PluginInfo(self,
+                            version='1.0.0',
+                            description='A plugin for testing.',
+                            author=Author('The Externals', 'tester@holmes.nl', 'NFI'),
+                            maturity=MaturityLevel.PROOF_OF_CONCEPT,
+                            webpage_url='https://hansken.org',
+                            matcher='file.extension=txt',
+                            id=PluginId(domain='nfi.nl', category='test', name='TestPlugin'),
+                            license='Apache License 2.0'
+                            )
+  ```
+
+## 0.3.0
+
+* Extraction Plugins can now create new datastreams on a Trace through data transformations. Data transformations
+  describe how data can be obtained from a source.
+
+  An example case is an extraction plugin that processes an archive file. The plugin creates a child trace per entry in
+  the archive file. Each child trace will have a datastream that is a transformation that marks the start and length of
+  the entry in the original archive data. By just describing the data instead of specifying the actual data, a lot of
+  space is saved.
+
+  Although Hansken supports various transformations, the Extraction Plugins SDK for now only supports ranged data
+  transformations. Ranged data transformations define data as a list of ranges, each range with an offset and length in
+  a bytearray.
+
+  The following example sets a new datastream with dataType `html` on a trace, by setting a ranged data transformation:
+
+  ```python
+  trace.add_transformation('html', RangedTransformation(Range(offset, length)))
+  ```
+
+  The following example creates a child trace and sets a new datastream with dataType `raw` on it, by setting a ranged
+  data transformation with two ranges:
+
+  ```python
+  child = trace.child_builder('new trace')
+  child.add_transformation('raw', RangedTransformation.builder()
+                                                      .add_range(10, 20)
+                                                      .add_range(50, 30)
+                                                      .build())
+  });
+  ```
+
+  More detailed documentation will follow in an upcoming SDK release.
+
+## 0.2.0
+
+.. warning:: This is an API breaking change.
+   Plugins created with an earlier version of the extraction plugin
+   SDK are not compatible with Hansken that uses `0.2.0` or later.
+
+* Introduced a new extraction plugin type `api.extraction_plugin.DeferredExtractioPlugin`.
+  Deferred Extraction plugins can be run at a different extraction stage.
+  This type of plugin also allows accessing other traces using the searcher.
+
+* The class `api.extraction_context.ExtractionContext` has been renamed to `api.data_context.DataContext`.
+  The new name `DataContext` represents the class contents better.
+  Plugins have to update matching import statements accordingly.
+  Plugins should also update the named argument `context` to `data_context` of the plugin `process()` method.
+  This change has no functional changes.
+
+  Old:
+
+  ```python
+  from hansken_extraction_plugin.api.extraction_context import ExtractionContext
+
+  def process(self, trace, context):
+    pass
+  ```
+
+  New:
+
+  ```python
+  from hansken_extraction_plugin.api.data_context import DataContext
+
+  def process(self, trace, data_context):
+    pass
+  ```
+
+* Moved `api.author.Author` to `api.plugin_info.Author`, and moved `api.maturity_level.MaturityLevel`
+  to `api.plugin_info.MaturityLevel`
+  This is a more *pythonic* way of grouping of classes into modules. This change has no functional side effects.
+
+  Plugins have to update matching import statements accordingly.
+
+  Old:
+
+  ```python
+  from hansken_extraction_plugin.api.author import Author
+  from hansken_extraction_plugin.api.maturity_level import MaturityLevel
+  from hansken_extraction_plugin.api.plugin_info import PluginInfo
+  ```
+
+  New:
+
+  ```python
+  from hansken_extraction_plugin.api.plugin_info import Author, MaturityLevel, PluginInfo
+  ```
+
+* Removed `DataContext.get_first_bytes()` from the public API.
+
+* Removed `api.extraction_trace.validate_update_arguments(..)` from the public API. This method is still invoked
+  implicitly when setting trace properties.

0.6.3/_sources/dev/python/debugging.md.txt

@@ -0,0 +1,162 @@
+# How to debug an Extraction Plugin
+
+Debugging is the art of removing bugs — hopefully quickly.
+
+## Locally
+
+To debug a plugin locally, it is recommended to start the plugin via the IDE. This has the advantage that breakpoints
+can easily be put in the code instead of printing log statements, for example. To start a plugin locally, a piece of
+code must be added, see [Testing](testing.md) for more information.
+
+### Logging
+
+The logging of the extraction plugin is displayed in the console.
+
+## Locally with Docker
+
+Debugging an extraction plugin via docker is a bit trickier. In order to debug in Python, a debugger must be added to
+the extraction plugin. There are several debug modules for Python available, but one debug module that works well with
+Visual Studio Code is [debugpy](https://github.com/microsoft/debugpy). This package is developed by Microsoft
+specifically for use in Visual Studio Code with Python.
+
+.. note:: `debugpy` implements the Debug Adapter Protocol (DAP), which is a standardised way for development tools to
+   communicate with debuggers.
+
+Using `debugpy` with Docker containers requires 4 distinct steps:
+
+1. Install `debugpy`
+2. Configuring `debugpy` in Python
+3. Build a docker image
+4. Configuring the connection to the Docker container
+5. Setting breakpoints in your code
+
+### Install `debugpy`
+
+First, add `debugpy` to your `setup.py`.
+
+```python
+from setuptools import setup
+
+setup(
+    # ...
+    install_requires=[
+        "hansken-extraction-plugin==0.4.7",  # the plugin SDK
+        "debugpy==1.5.1"
+    ]
+)
+```
+
+### Configuring `debugpy` in Python
+
+At the beginning of your script, import `debugpy`, and call `debugpy.listen()` to start the debug adapter, passing
+a `(host, port)` tuple as the first argument. Use the `debugpy.wait_for_client()` function to block program execution
+until the client is attached.
+
+```python
+import debugpy
+
+debugpy.listen(("0.0.0.0", 5678))
+debugpy.wait_for_client()  # blocks execution until client is attached
+
+# your extraction plugin code
+```
+
+### Build a Docker image
+
+If the Docker image is not built, first build the image as described 
+[here](getting_started.md#Building a docker image for a Python plugin).
+
+
+### Configuring the connection to the Docker container
+
+`debugpy` is now set up to accept connections inside a Docker container. To connect to `debugpy` in the docker
+container, port 5678 must be published. To make a port available to services outside of Docker, use the --publish or -p
+flag. This creates a firewall rule which maps a container port to a port on the Docker host to the outside world.
+
+To run the extraction plugin with the published port the following command can be used:
+
+```bash
+docker run -p 5678:5678 your_extraction_plugin_name
+```
+
+The next step is to configure Visual Studio Code. A `launch.json` file must be created in order for Visual Studio Code
+to connect to the extraction plugin in Docker. This minimal `launch.json` example below tells the debugger to attach
+to `localhost` on port `5678`.
+
+```json
+{
+  "version": "0.2.0",
+  "configurations": [
+    {
+      "name": "Python: Remote Attach",
+      "type": "python",
+      "request": "attach",
+      "connect": {
+        "host": "localhost",
+        "port": 5678
+      },
+      "pathMappings": [
+        {
+          "localRoot": "${workspaceFolder}",
+          "remoteRoot": "."
+        }
+      ]
+    }
+  ]
+}
+```
+
+### Setting breakpoints in the code
+
+The last step is to add breakpoints in the code.
+
+### Logging in Docker
+
+The logging of the extraction plugin is displayed in the console after running the `docker run` command. In addition,
+the logging is also displayed in the Visual Studio Code console while debugging.
+
+## Kubernetes
+
+In kubernetes it is currently _not_ possible to debug via `debugpy` because no debug ports are published.
+
+### Logging in Kubernetes
+
+If there is authorization to the kubernetes cluster, the logging can be viewed with the following command:
+
+```bash
+kubectl logs -f hansken-extraction-plugins/your_extraction_plugin_pod
+```
+
+## Debug HQL
+
+An HQL query can be debugged by running the test framework with the `--verbose` option enabled. The found HQL matches
+will then be displayed in the console. To test a plugin in python with the `--verbose` option enabled use the following
+command:
+
+```bash
+test_plugin --standalone plugin/your_plugin.py --regenerate --verbose
+```
+
+The following output will then be displayed in the console:
+
+```text
+HQL match found for:
+$data.type=jpg
+With trace:
+dataType=jpg
+types={file, data}
+properties={data.raw.mimeType=image/jpg, path=/test-input-trace, file.name=image.jpg, name=test-input-trace, id=0}
+```
+
+If the HQL query contains an error, it will be shown in the generated test results. An example of an invalid query
+is ``$data.mimeType=image/jpg`` (slash not escaped). This query will produce an error like the one shown below.
+
+```json
+{
+  "class": "org.hansken.plugin.extraction.hql_lite.lang.ParseException",
+  "message": "HqlLiteHumanQueryParser: line 1:20 token recognition error at: '/jpg'"
+}
+```
+
+.. note:: The error is only shown in the generated trace, so to find out the `ParseException` run the ``test_plugin``
+   command with the ``--regenerate`` option enabled.

0.6.3/_sources/dev/python/getting_started.md.txt

@@ -0,0 +1,195 @@
+# Getting started
+
+Set up a development environment: step by step.
+
+The following section describes how to set up a fully working development environment for extraction plugins with Python.
+This is written for those who are not comfortable setting up a working build environment.
+This is **optional**; advanced users may choose a different development environment setup, and can skip this section completely.
+
+If you fail to set up a development environment, feel free to ask for help at our Discord channel. 
+
+
+## Install required software on Ubuntu
+
+In order to be able to develop Hansken Extraction Plugins in Python on Ubuntu, the following build tools need be installed on your system: python, pip, tox, Java, Docker.
+
+* **Python, pip, tox, Java**  
+  To install, run the following commands in your terminal:
+ 
+  ```bash
+  sudo apt update
+  sudo apt install python3.8 python3-pip tox default-jdk
+  ```
+  
+  You can check whether all versions are installed correctly by running the following commands (and validate the output):
+  ```bash
+  python --version
+  # should return version 3.8.10 or higher
+  
+  pip3 --version
+  # should return version 20.0.2 or higher
+  
+  tox --version
+  # should return version 3.11.2 or higher
+  
+  java -version
+  # should return version 11.0.4 or higher
+  ```
+
+If the above software is installed, you can continue with "Install Docker",
+or if you don't want to install Docker, continue with "Install your IDE: Pycharm".
+
+
+## Install required software on Windows.
+
+In order to be able to develop Hansken Extraction Plugins in Python on Windows, follow the next steps:
+
+* For verification of the installation of each program we will use commands on the command prompt. 
+  This can be opened by clicking the Windows Start button and typing:
+  
+  ```bash
+  cmd
+  ```
+  
+  Then hit Enter . This will open the command prompt where you can enter the commands given in the following steps.
+
+* **Python 3.8 or higher & pip**
+
+  Download the installer from [python.org/Downloads](https://www.python.org/downloads/) (click the yellow "Download Python" button)
+  and run it to install Python and pip. 
+  Pip is the standard package manager for Python. 
+  It allows you to install and manage additional packages that are not part of the Python standard library, like the Extraction Plugins SDK.
+  
+  Be sure to select the option "Add python to PATH".
+  
+  When the installation is complete, verify the installation by checking the Python and pip versions: 
+  
+  ```bash
+  python --version
+  # should return your downloaded Python version (>3.8.5)
+  
+  pip3 --version
+  # should return 20.2.3 or higher
+  ```
+
+* **tox**  
+  Tox is used to automate and standardize testing in Python. Use Pip to install Tox with this command:
+
+  ```bash
+  pip install tox
+  ```
+  
+  Verify that Tox is installed by running: 
+  
+  ```bash
+  tox --version
+  # should return 3.23.0 or higher
+  ```
+
+* **Java JDK 11.0.4 (or higher)**  
+  Java JDK 11.0.4 or higher is needed to run the test framework of the Extraction Plugins SDK.
+  This enables you to test without actually deploying the plugin in Hansken.
+  Installing Java on Windows can be done in many different way, but for now we can not recommend one method.
+  Please have a look at [Install the Microsoft Build of OpenJDK](https://docs.microsoft.com/en-us/java/openjdk/install) for more details.  
+
+  ```bash
+  java -version
+  # should return 11.0.4 or higher
+  ```
+  
+  N.b. Make sure to set the environment variable in case the installed JDK is not detected by the system. 
+  You can follow the below-mentioned steps to set the environment variable:
+  
+  1. Click the Windows start button
+  2. Type "advanced system settings"
+  3. Hit enter
+  4. Now click on Environment Variables, select Path under System Variables section and click on Edit. We need to add the path of installed JDK to system Path.
+  5. Click on New Button and add the path to installed JDK bin which is C:\java\java-11\jdk-11.0.4\bin in our case.
+  6. Press OK Button 3 times to close all the windows. This sets the JDK 11 on system environment variables to access the same from the console.
+  
+
+If the above software is installed, you can continue with "Install Docker",
+or if you don't want to install Docker, continue with "Install your IDE: Pycharm".
+
+## Install Docker (Ubuntu, Windows)
+Installing Docker is a bit more complicated. The Docker website describes the installation instructions in detail.
+Please follow the instructions on [docs.docker.gom/get-docker/](https://docs.docker.com/get-docker/) to install docker.
+
+* To check you have Docker installed correctly, run
+
+  ```bash
+  docker --version
+  # should return version 20.10.17 or higher
+  ```
+
+Note: if you run Docker inside a managed network, you might also need to configure proxies and/or certificates.
+Please contact your system administrator if you need help with this.
+
+
+## Set up your IDE: PyCharm
+
+We recommend that you use an IDE to aid you in your development of Extraction Plugins. PyCharm is a good choice.
+
+* JetBrains has an excellent installation guide on their webpage: [Click here to go to the Pycharm Installation Guide](https://www.jetbrains.com/help/pycharm/installation-guide.html)
+
+
+## Download an extraction plugin template (empty plugin)
+You can download an extraction plugin template.
+This is an empty plugin, from which you can rapidly start your plugin development.
+
+* The template is hosted on GitHub: https://github.com/NetherlandsForensicInstitute/hansken-extraction-plugin-template-python.
+  You can download a zip with the template from here. The below screenshot shows where the download button is located:
+
+  ![](getting_started/download_template.png)
+
+
+## Import the Extraction Plugins Skeleton in PyCharm
+
+* First unzip the skeleton plugin downloaded from the previous step.
+* Next, start PyCharm.
+* When PyCharm starts, choose "Open" and select the folder where you placed the Extraction Plugin Skeleton.
+![](getting_started/pycharm_open_project.png)
+
+* The following popup will appear, click OK .
+![](getting_started/pycharm_open_project_venv.png)
+
+* The Extraction Plugins Skeleton is now loaded in PyCharm, which should look as follows:
+![](getting_started/pycharm_project_start_page.png)
+
+* Be sure to give the `README.md` a read when you are done with the Prerequisites.
+
+
+## Verify full setup
+To verify that your system has been setup correctly, you can run the test suite in the Extraction Plugin Skeleton:
+
+* First, press `alt-F12` at the same time to open a terminal in PyCharm in the project root folder.
+* To run the tests of the Skeleton, run this command in the terminal from the root folder:
+  ```bash
+  tox
+  ```
+  The first time running tox may take a few minutes! Please be patient 😊
+
+  Tox will install all required plugin dependencies, and start your tests.
+  N.b. The plugin template demonstrates how to build a plugin with tox.
+  Some plugin developers choose a different tool than tox.
+
+* If your system has been set up correctly, the output should end with a summary like this:
+
+  ```
+  py38: commands succeeded
+  congratulations :)
+  ```
+  
+  This means the setup is finished. You now have everything installed to start coding your own plugin!
+
+
+## Next steps
+
+Now that you have a working environment, you can start doing cool stuff.
+Please have a look at the following pages for more information:
+
+* [Packaging](packaging.md): how to package your plugin and use it in Hansken
+* [Run plugins with Hansken.py](hanskenpy.md): run your plugin on a case without uploading the plugin to Hansken, useful for quick prototyping
+* [Testing](testing.md): how to write tests for your plugin
+* [Debugging](debugging.md): if your plugin isn't working as expected, you can debug it
+* [Snippets](snippets.md): code snippets to demonstrate common plugin usage patterns

0.6.3/_sources/dev/python/hanskenpy.md.txt

@@ -0,0 +1,91 @@
+# Run plugins with Hansken.py
+
+Hansken.py is a Python client to Hansken's REST API, developed and maintained by the Netherlands Forensic Institute.
+
+With Hansken.py, you can run your Python plugin on a project on your Hansken installation that has already been
+extracted. This way of running your plugin is useful during your plugin development. It is not required to upload your
+plugin to Hansken and start an extraction, making the development cycle faster. However, please note that this is only
+useful during the development stage of your plugin, as the execution of your plugin will be much slower compared to
+running the plugin during a Hansken extraction.
+
+.. note:: The execution of your plugin will be much slower in Hansken.py compared to running the plugin during a Hansken
+   extraction.
+
+## How to run python extraction plugins standalone with Hansken.py
+
+Running python extraction plugins standalone with Hansken.py is easy. It is just one command. This section explains how
+to setup this command for your specific environment.
+
+### Create a runner file
+
+Create a file `run_with_hansken.py` in the root folder of your plugin. This will aid you in running the plugin with
+hansken.py.
+
+```python
+from hansken_extraction_plugin.runtime.extraction_plugin_runner import run_with_hanskenpy
+
+from plugin.my_plugin import MyPlugin
+
+if __name__ == '__main__':
+    run_with_hanskenpy(MyPlugin)
+```
+
+### Preparing for the command
+
+Before you can enter the command that runs your extraction plugin with Hansken.py, you need to find three values:
+
+* `HANSKEN_PROJECT_ID` project id on which you want to run your plugin
+* `YOUR_GATEKEEPER_URL` the URL to the Hansken gatekeeper
+* `YOUR_KEYSTORE_URL` the URL to your keystore
+
+The correct values of these variables can be found in the Expert UI. Go to the search-page of your project in the
+ExpertUI. Next to the search-bar hit the button "Save query".
+
+![hanskenpy_save_query.png](hanskenpy_save_query.png)
+
+A new dialog shows up. At the bottom of this dialog, you will find your gatekeeper and keystore urls as well as the
+project id.
+
+![hanskenpy_gatekeeper_keystore.png](hanskenpy_gatekeeper_keystore.png)
+
+### Running your plugin with Hansken.py
+
+Next, open a terminal in the project root folder of your plugin and enter the following command. It will run your
+extraction plugin with Hansken.py in Hansken. Replace the three variables with their respective values.
+
+```bash
+python3 ./run_with_hanskenpy.py -v -l - HANSKEN_PROJECT_ID --endpoint YOUR_GATEKEEPER_URL --keystore YOUR_KEYSTORE_URL
+```
+
+If your command runs well, you might be prompted for your username and password. There will be some output (note that
+the output may vary depending on your system setup and project content):
+
+```text
+[2021-03-16 12:59:45.344248+0000] INFO: hansken.auth: selected IDP ID (...) with SOAP endpoint (...)
+[2021-03-16 12:59:45.344450+0000] WARNING: hansken.auth: IDP url known, user+pass auth required but no username supplied
+username []: testaccount
+[2021-03-16 12:59:48.423245+0000] INFO: hansken.auth: user acknowledged environment username or supplied custom username: testaccount
+password for user testaccount:
+[2021-03-16 12:59:53.799668+0000] INFO: hansken.auth: identity provider url and user+pass provided or known, using Keycloak SAML with Basic auth
+[2021-03-16 12:59:53.805538+0000] INFO: hansken_extraction_plugin.runtime.extraction_plugin_runner: PluginRunner is running plugin class Plugin
+[2021-03-16 12:59:53.859299+0000] INFO: hansken.auth: posting SAML request with authorization for user testaccount to IDP endpoint (...)
+[2021-03-16 12:59:54.240290+0000] INFO: plugin.extraction_plugin: processing trace 54197e67-8135-40c3-93f1-3d73a5552693
+[2021-03-16 12:59:54.240753+0000] INFO: plugin.extraction_plugin: processing trace OCRimage
+[2021-03-16 12:59:54.240753+0000] INFO: plugin.extraction_plugin: processing trace (...)
+```
+
+Note that the arguments `-v` and `-l -` are passed to enable logging. To find out what other options can be passed to
+this command, please have a look at the hansken.py documentation, or simply run the following command:
+
+```bash
+python3 ./run_with_hanskenpy.py --help
+```
+
+## Compatibility
+
+At this moment, running Extraction Plugins with Hansken.py has a few limitations. These are:
+
+* When writing an Extraction Plugin for use with Hansken.py, the matcher must contain exactly one "$data.property =
+  value" expression.
+* [Data transformations](../concepts/data_transformations.md) are currently not supported by Hansken.py.
+* :ref:`Tracelets<tracelets>` are not yet supported by the SDK in use with Hansken.py.

0.6.3/_sources/dev/python/packaging.md.txt

@@ -0,0 +1,43 @@
+# Packaging
+
+Extraction plugins are packaged as OCI images (also known as Docker images).
+The OCI images are _labeled_ with the PluginInfo.
+To automate packaging of a Python plugin and labeling the OCI image,
+the Extraction Plugin SDK comes with a utility application `build_plugin`.
+
+Make sure that the Extraction Plugins SDK is [installed](getting_started.md#Installation) as well as Docker.
+
+Then, build your plugin container image using the following command:
+
+```bash
+build_plugin PLUGIN_FILE DOCKER_FILE_DIRECTORY [DOCKER_IMAGE_NAME] [DOCKER_ARGS]
+```
+
+For example:
+```bash
+build_plugin chatplugin.py . chatplugin --build-arg http_proxy="$http_proxy" --build-arg https_proxy="$https_proxy"
+```
+
+This will generate a plugin image:
+* The extraction plugin is added to your local image registry (`docker images`),
+* Note that the variables `$http_proxy` and `$https_proxy` are put in quotes, this is needed in case they contain
+  spaces,
+* The image is tagged with two tags: `latest`, and your plugin version.
+
+Arguments:
+* PLUGIN_FILE: Path to the python file of the plugin.
+* DOCKER_FILE_DIRECTORY: Path to the directory containing the Dockerfile of the plugin.
+* (Optional) [DOCKER\_IMAGE\_NAME]: Name of the docker image without tag. Note that docker image names cannot start with
+  a period or dash. If it starts with a dash, it will be interpreted as an additional docker argument (see
+  DOCKER\_ARGS). If no name is given the name defaults to `extraction-plugin/PLUGINID`, e.g.
+  `extraction-plugin/nfi.nl/extract/chat/whatsapp`.
+* (Optional) [DOCKER\_ARGS]: Additional arguments for the docker command, which can be as many arguments as you like.
+
+To verify that the image has been built, type the following command to view all local images:
+```bash
+docker images
+```
+
+Once your plugin is packaged, it can be published or 'uploaded' to Hansken.
+See ":ref:`upload_plugin`" for instructions.
+

0.6.3/_sources/dev/python/prerequisites.md.txt

@@ -0,0 +1,9 @@
+# Prerequisites
+
+All required project dependencies to build extraction plugins are published on the public [PyPI](https://pypi.org/project/hansken-extraction-plugin/).
+
+Required:
+
+* Python 3.8 or higher
+* Java 11 (for running the test-framework, which is implemented in Java)
+* Docker (for packaging and deploying extraction plugins in containers, can be omitted if you have an external build pipeline that provides Docker)

0.6.3/_sources/dev/python/snippets.md.txt

@@ -0,0 +1,220 @@
+# Python code snippets
+
+## Adding properties to a trace
+
+Use :py:meth:`update <hansken_extraction_plugin.api.extraction_trace.ExtractionTraceBuilder.update>`
+to add trace types and their properties to an
+:py:class:`ExtractionTrace <hansken_extraction_plugin.api.extraction_trace.ExtractionTrace>`.
+Example:
+
+```python
+def process(self, trace, data_context):
+    # get the name of the file
+    file_name = trace.get('file.name')
+    # set the chat application property on the trace
+    trace.update('chatConversation.application', f'DemoApp {file_name}')
+```
+
+All types and properties that can be set are defined in the :ref:`Hansken trace model`.
+
+### Date properties
+
+When adding a property which holds a value of data-type Date, always define timezone as being UTC. Example:
+
+```python
+def process(self, trace, data_context):
+    trace.update('file.modifiedOn',
+                 datetime.fromtimestamp(1630510809, tz=timezone.utc))
+```
+
+
+### Category for extra properties
+
+If the information, which must be added as a property, does not match any of the existing properties of Hansken trace
+model, use the category "misc" (miscellaneous). When part of the category "misc", any name can be given to a property.
+The values of miscellaneous properties are expected to be of data-type string. Example:
+
+```python
+def process(self, trace, data_context):
+    trace.update({
+        'file.misc.notes': 'Some additional notes about the file trace.',
+        'file.misc.anyName': 'Even more notes.'
+    })
+```
+
+.. _tracelets python:
+
+
+### Adding tracelets
+
+In the following Python example, a "prediction" :ref:`tracelet<tracelets>` is added to a trace. The tracelet consists
+of a list of four properties, namely "class", "confidence", "modelName" and "modelVersion".
+
+```python
+trace.add_tracelet(Tracelet('prediction', {'class': 'telephone',
+                                           'confidence': 0.8,
+                                           'modelName': 'yolo',
+                                           'modelVersion': '2.0'}))
+```
+
+
+## Adding child traces to a trace
+
+Adding child traces to the trace can be done by creating a builder with
+:py:meth:`child_builder <hansken_extraction_plugin.api.extraction_trace.ExtractionTraceBuilder.child_builder>`.
+Example:
+
+```python
+def process(self, trace, data_context):
+    child_builder = trace.child_builder('childTrace-1')
+    child_builder.update({
+        'chatMessage.application': 'DemoApp',
+        'chatMessage.from': 'Ann',
+        'chatMessage.to': ['Mark'],
+        # list, because there can be multiple receivers
+        'chatMessage.message': 'Hello, are you there?',
+    }).build()
+    grandchild_builder = child_builder.child_builder('grandchild')
+    grandchild_builder.update(data={'byte': b'some bytes'})
+    grandchild_builder.build()
+```
+
+This adds a single child trace with name `childTrace-1` and four properties, as well as a grandchild trace with name 
+`grandchild` and a byte data stream.
+
+.. _datastreams python:
+
+
+## Adding data to a trace
+
+Traces can have data attached to them. See :ref:`datastreams` for more information.
+The following two snippets demonstrate how to add data to a trace.
+
+It is currently not possible to verify that a specific data stream is already set or not.
+
+
+### Data Transformations
+
+The most efficient way to add data to a trace is using data transformations.
+See :doc:`../concepts/data_transformations` for more details.
+
+The following example sets a new datastream with dataType `html` on a trace, by setting a ranged data transformation:
+
+```python
+trace.add_transformation('html', RangedTransformation(Range(offset, length)))
+```
+
+The following example creates a child trace and sets a new datastream with dataType `raw` on it, by setting a ranged
+data transformation with two ranges:
+
+```python
+child = trace.child_builder('new trace')
+child.add_transformation('raw', RangedTransformation.builder()
+                         .add_range(10, 20)
+                         .add_range(50, 30)
+                         .build())
+});
+```
+
+
+### Blobs
+
+It is not always possible to create a transformation for the data that has to be
+added to a trace. For example, if the data is a result of a computation, and not
+a direct subset of another data stream..
+
+The following snippet shows how to create a new data stream of dataType `raw` on a trace from a blob stored in `bytes`:
+
+```python
+data = {'raw': b'...'}
+trace.update(data=data);
+```
+
+
+## Specifying system resources
+
+It is possible to specify **maximum** system resources in the `PluginInfo`. To run a plugin with 0.5 cpu (= 0.5
+vCPU/Core/hyperthread) and 1 gb memory, for example, the following configuration can be added to `PluginInfo`:
+
+```python
+plugin_info = PluginInfo(...,
+                         resources=PluginResources(maximum_cpu=0.5, maximum_memory=1000))
+```
+
+
+.. _python_snippets_deferred:
+
+## Deferred Plugins
+
+Implementing a deferred extraction plugin requires inheriting the
+:py:class:`DeferredExtractionPlugin <hansken_extraction_plugin.api.extraction_plugin.DeferredExtractionPlugin>`
+base class.
+
+```python
+class DeferredPlugin(DeferredExtractionPlugin):
+    def process(self, trace, context, searcher):
+```
+
+This allows accessing a third :py:class:`TraceSearcher <hansken_extraction_plugin.api.trace_searcher.TraceSearcher>`
+parameter in the process function. This can be used to search for traces:
+
+```python
+with searcher.search('file.extension:html', 10) as searchresult:
+    for trace in searchresult:
+        log.debug(f'extension {trace.get("file.extension")}')
+```
+
+The search method accepts two arguments; a HQL query and the maximum number of traces the return. The ``search`` method
+accepts an HQL query and a count, which represents the maximum number of traces to return.
+
+It may be useful to specifically search for traces from the image being extracted. Add ``"image:" + trace.get("image")``
+to your query. The query of the provided example could be extended like
+this: `"file.extension:html AND image:" + trace.get("image")`.
+
+The returned :py:class:`SearchResult <hansken_extraction_plugin.api.search_result.SearchResult>`
+should be closed, for example by using `with`. The resulting search result is an iterable, which will be exhausted when
+no more traces are available. The search result allows taking one or more traces by calling :py:
+meth:`take <hansken_extraction_plugin.api.search_result.SearchResult.take>` or
+:py:meth:`takeone <hansken_extraction_plugin.api.search_result.SearchResult.takeone>`.
+
+
+## Logging
+
+We use Logbook to log messages in Python. Logbook is a logging system for Python that replaces the standard library’s
+logging module.
+
+To enable logging in your plugin, add the following to the top of your plugin code:
+
+```python
+from logbook import Logger
+
+log = Logger(__name__)
+```
+
+From there on the logging is pretty straight forward:
+
+```python
+log.info(f'Logging a variable: {my_variable}')
+```
+
+The default log level is WARNING. You can use the `-v` (or `-vv` or `-vvv`) option of `serve_plugin.py` to increase the
+log level. This is typically done in the plugin `Dockerfile`.
+
+.. warning:: Be careful with logging sensitive information.
+
+.. note:: Contact your Hansken administrator for more information on where to find logs for your Hansken environment.
+
+## [EXPERIMENTAL FEATURE] Adding previews to a trace
+
+.. warning:: This is an experimental feature, which might change or get removed in future releases.
+
+Use :py:meth:`update <hansken_extraction_plugin.api.extraction_trace.ExtractionTraceBuilder.update>`
+to add previews to an
+:py:class:`ExtractionTrace <hansken_extraction_plugin.api.extraction_trace.ExtractionTrace>`.
+Example:
+
+```python
+def process(self, trace, data_context):
+    # set the preview data for the image/png MIME-type
+    trace.update('preview.image/png', b'\x00\xff')
+```

0.6.3/_sources/dev/python/testing.md.txt

@@ -0,0 +1,108 @@
+# Advanced use of the Test Framework in Python
+
+.. _python testing:
+
+This section assumes you use the same setup as is used in
+the [Extraction Plugin Examples](https://git.eminjenv.nl/hanskaton/hansken-extraction-plugin-sdk/examples).
+
+By default, the build scripts as described in the [Getting Started](getting_started.md) section will automatically run
+tests. The appropriate commands have been added to the tox.ini directly. This section gives a little more detail on the
+test commands and options.
+
+One can simply create unit tests for a plugin directly. However, we also provide a test-framework for testing them over
+gRPC. The test-framework serves a running instance of a Python plugin, and feeds it input files and compares the results
+against an expected result set.
+
+Note that the test-framework is implemented in Java, hence the Java 11 requirement. A jar file is included in the Python
+SDK which is called from a Python wrapper.
+
+## Regenerate expected test results
+
+The build and test scripts run some integration tests. To update the expected test outcome, the following command can be
+used:
+
+```bash
+tox -e regenerate
+```
+
+## Standalone testing
+
+The test runner is a script called `test_plugin` which is available in the SDK.
+
+To get started, `cd` into the directory of the plugin you want to test and run:
+
+```bash
+test_plugin --standalone plugin/chat_plugin.py
+```
+
+Note that the argument provided to the option `--standalone` must be the relative path to the plugin `py` file which is
+to be tested. This test accepts input files from the directory `testdata/input` and compares the results to the result
+files in found in `testdata/results`. Use the optional argument `--regenerate` to regenerate the expected results for
+the test when needed.
+
+This standalone test is also used by the `tox.ini` file to validate the plugin. Simply calling `tox` should be enough to
+install all dependencies and run the test.
+
+## Testing with a Docker image
+
+If there is a docker image available for the plugin you can also test it by executing:
+
+```bash
+test_plugin --docker extraction-plugin-examples-my-plugin
+```
+
+Replace the 'extraction-plugin-examples-chat' with the docker image you want to test. Run the following command to see
+which docker images are available:
+
+```bash
+docker images
+```
+
+## Manual testing
+
+The third option for testing is a manually started plugin. Start the plugin service in a terminal by executing:
+
+```bash
+serve_plugin -vvv plugin/my_plugin.py
+```
+
+This will spin up the chat plugin at port 8999. Here also the argument must be a path to the plugin's `.py` file. In
+another terminal window, run the test with:
+
+```bash
+test_plugin --manual localhost 8999
+```
+
+## Tip: Start tests in your IDE
+To start the extraction plugin from code, create a `__main__` method which calls the `_test_validate_standalone`
+method of the test framework (see the example below). This method causes the extraction plugin to be started and
+supplied with data by the FLITS test framework. In this way the test can be started from the IDE, which has the
+advantage that it is easier to debug.
+
+```python
+from hansken_extraction_plugin.test_framework.test_plugin import _test_validate_standalone
+from hansken_extraction_plugin.api.extraction_plugin import ExtractionPlugin
+
+
+class PluginToTest(ExtractionPlugin):
+
+    def plugin_info(self):
+        # return plugin info
+        pass
+
+    def process(self, trace, data_context):
+        # process the data/trace here
+        pass
+
+
+if __name__ == '__main__':
+    _test_validate_standalone(PluginToTest, 'testdata/input', 'testdata/result', False)
+```
+
+## Help
+
+Run the following for an overview of all the available options in the test script:
+
+```bash
+test_plugin --help
+```