mirror of
https://github.com/NetherlandsForensicInstitute/hansken-extraction-plugin-sdk-documentation.git
synced 2026-08-24 20:17:16 +00:00
0.6.3
This commit is contained in:
remco
@@ -0,0 +1,205 @@
|
||||
# Java API Changelog
|
||||
|
||||
This document summarizes all important API changes in the Extraction Plugin API. This document only shows changes that
|
||||
are important to plugin developers. For a full list of changes per version, please refer to the general
|
||||
:ref:`changelog <changelog>`.
|
||||
|
||||
.. If present, remove `..` before `## |version|` if you create a new entry after a previous release.
|
||||
|
||||
.. ## |version|
|
||||
|
||||
|
||||
## 0.6.1
|
||||
|
||||
* The JAVA SDK is now distributed through maven central instead of the Hansken community.
|
||||
|
||||
## 0.6.0
|
||||
|
||||
.. warning:: It is highly recommended to upgrade your plugin to this new version.
|
||||
See the migration steps below.
|
||||
|
||||
* Extraction plugin container images are now labeled with PluginInfo. This
|
||||
allows Hansken to efficiently load extraction plugins.
|
||||
|
||||
* By default, extraction plugin version is managed in the plugin's `pom.xml`.
|
||||
The `.pluginVersion(..)` can be removed from the PluginInfo builder.
|
||||
|
||||
* **Migration steps from earlier versions** -- for plugins that use the Java
|
||||
extraction plugin SuperPOM:
|
||||
|
||||
1. Update the SDK version in your `pom.xml`
|
||||
2. If you come from a version prior to `0.4.0`, or if you use a plugin name
|
||||
instead of a plugin id in your `pluginInfo()`, switch to the plugin id style
|
||||
(read instructions for version `0.4.0`)
|
||||
3. Set your plugin version in your project's `pom.xml`, and remove the
|
||||
following from your `PluginInfo.Builder`:
|
||||
|
||||
```java
|
||||
.pluginVersion(...)
|
||||
```
|
||||
|
||||
4. Update your build scripts to build your plugin (Docker) container image.
|
||||
You should build your plugin container image with the following command:
|
||||
|
||||
```bash
|
||||
mvn package docker:build`
|
||||
```
|
||||
|
||||
This will generate a plugin image:
|
||||
|
||||
* The extraction plugin is added to your local image registry
|
||||
(`docker images`),
|
||||
* The image name is `extraction-plugin/PLUGINID`, e.g.
|
||||
`extraction-plugin/nfi.nl/extract/chat/whatsapp`,
|
||||
* The image is tagged with two tags: `latest`, and your plugin version.
|
||||
|
||||
Nb. If Docker is not available in your environment, `podman` can be used
|
||||
as an alternative. See :ref:`packaging <java_superpom_podman>` for more
|
||||
details.
|
||||
|
||||
## 0.5.0
|
||||
|
||||
* Add new tracelet api `Trace.addTracelet(type, consumer)`.
|
||||
It can be used like this:
|
||||
|
||||
```java
|
||||
trace.addTracelet("prediction", tracelet -> tracelet
|
||||
.set("type", "classification")
|
||||
.set("label", "label")
|
||||
.set("confidence", 0.8f)
|
||||
.set("embedding", Vector.of(1,2,3))
|
||||
.set("modelName", "yolo")
|
||||
.set("modelVersion", "2.0"));
|
||||
```
|
||||
|
||||
* Deprecate Trace.addTracelet(Trace)
|
||||
* Support vector data type in trace properties.
|
||||
|
||||
## 0.4.13
|
||||
|
||||
* When writing input search traces for tests, it is no longer required to explicitly set an `id` property.
|
||||
These are automatically generated when executing tests.
|
||||
|
||||
## 0.4.7
|
||||
|
||||
* A new convenience method `id(String, String, String)` is added to the PluginInfo builder. This removes some
|
||||
boilerplate code when setting the pluginId. More details on the plugin naming conventions can be found at the
|
||||
:doc:`../concepts/plugin_naming_convention` section.
|
||||
|
||||
```java
|
||||
PluginInfo.builderFor(this)
|
||||
.id("nfi.nl", "extract", "TestPlugin") // new style
|
||||
.id(new PluginId("nfi.nl", "extract", "TestPlugin")) // old style, but also works
|
||||
...
|
||||
```
|
||||
|
||||
## 0.4.6
|
||||
|
||||
* It is now possible to specify maximum system resources in the `PluginInfo`. To run a plugin with 0.5 cpu (= 0.5
|
||||
vCPU/Core/hyperthread) and 1 gb memory, for example, the following configuration can be added to `PluginInfo`:
|
||||
|
||||
```java
|
||||
PluginInfo.builderFor(this)
|
||||
...
|
||||
.pluginResources(PluginResources.builder()
|
||||
.maximumCpu(0.5f)
|
||||
.maximumMemory(1000)
|
||||
.build())
|
||||
.build();
|
||||
```
|
||||
|
||||
## 0.4.0
|
||||
|
||||
* Extraction Plugins are now identified with a `PluginInfo.PluginId` containing a domain, category and name. The
|
||||
method `PluginInfo.name(pluginName)` has been replaced by `PluginInfo.id(new PluginId(domain, category, name)`. More
|
||||
details on the plugin naming conventions can be found at the :doc:`../concepts/plugin_naming_convention` section.
|
||||
|
||||
* `PluginInfo.name()` is now deprecated (but will still work for backwards compatibility).
|
||||
|
||||
* A new license field `PluginInfo.license` has also been added in this release.
|
||||
|
||||
* The following example creates a PluginInfo for a plugin with the name `TestPlugin`, licensed under
|
||||
the `Apache License 2.0` license:
|
||||
|
||||
```java
|
||||
PluginInfo.builderFor(this)
|
||||
.id(new PluginId("nfi.nl", "extract", "TestPlugin")) // id.domain: nfi.nl, id.category: extract, id.name: TestPlugin
|
||||
// .name("TestPlugin") // no longer supported
|
||||
.pluginVersion("0.4.1")
|
||||
.author(Author.builder()...build())
|
||||
.description("A plugin for testing.")
|
||||
.maturityLevel(MaturityLevel.PROOF_OF_CONCEPT)
|
||||
.hqlMatcher("*")
|
||||
.webpageUrl("https://www.hansken.org")
|
||||
.license("Apache License 2.0")
|
||||
.build();
|
||||
```
|
||||
|
||||
## 0.3.0
|
||||
|
||||
* Extraction Plugins can now create new datastreams on a Trace through data transformations. Data transformations
|
||||
describe how data can be obtained from a source.
|
||||
|
||||
An example case is an extraction plugin that processes an archive file. The plugin creates a child trace per entry in
|
||||
the archive file. Each child trace will have a datastream that is a transformation that marks the start and length of
|
||||
the entry in the original archive data. By just describing the data instead of specifying the actual data, a lot of
|
||||
space is saved.
|
||||
|
||||
Although Hansken supports various transformations, the Extraction Plugins SDK for now only supports ranged data
|
||||
transformations. Ranged data transformations define data as a list of ranges, each range with an offset and length in
|
||||
a bytearray.
|
||||
|
||||
The following example sets a new datastream with dataType `html` on a trace, by setting a ranged data transformation:
|
||||
|
||||
```java
|
||||
trace.setData("html", RangedDataTransformation.builder().addRange(offset, length).build());
|
||||
```
|
||||
|
||||
The following example creates a child trace and sets a new datastream with dataType `raw` on it, by setting a ranged
|
||||
data transformation with two ranges:
|
||||
|
||||
```java
|
||||
trace.newChild(format("lineNumber %d", lineNumber), child -> {
|
||||
child.setData("raw", RangedDataTransformation.builder()
|
||||
.addRange(10, 20)
|
||||
.addRange(50, 30)
|
||||
.build());
|
||||
});
|
||||
```
|
||||
|
||||
More detailed documentation will follow in an upcoming SDK release.
|
||||
|
||||
## 0.2.0
|
||||
|
||||
.. warning:: This is an API breaking change. Plugins created with an earlier version of the extraction plugin SDK are
|
||||
not compatible with Hansken that uses `0.2.0` or later.
|
||||
|
||||
* Introduced a new extraction plugin type `DeferredExtractioPlugin`. Deferred Extraction plugins can be run at a
|
||||
different extraction stage. This type of plugin also allows accessing other traces using the searcher.
|
||||
|
||||
* The class `ExtractionContext` has been renamed to `DataContext`. The new name `DataContext` represents the class
|
||||
contents better. Plugins have to update matching import statements and the type in `ExtractionPlugin.process()`
|
||||
implementation in the same way. This change has no functional side effects.
|
||||
|
||||
Old:
|
||||
|
||||
```java
|
||||
import org.hansken.plugin.extraction.api.ExtractionContext;
|
||||
|
||||
@Override
|
||||
|
||||
public void process(final Trace trace, final ExtractionContext context) throws IOException {
|
||||
|
||||
}
|
||||
```
|
||||
|
||||
New:
|
||||
|
||||
```java
|
||||
import org.hansken.plugin.extraction.api.DataContext;
|
||||
|
||||
@Override
|
||||
public void process(final Trace trace, final DataContext dataContext) throws IOException {
|
||||
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,134 @@
|
||||
# How to debug an Extraction Plugin
|
||||
|
||||
Debugging is the art of removing bugs — hopefully quickly.
|
||||
|
||||
## Locally
|
||||
|
||||
To debug a plugin locally, it is recommended to start the plugin via the IDE by running the integration test. This has
|
||||
the advantage that breakpoints can easily be put in the code instead of printing log statements, for example.
|
||||
|
||||
### Logging
|
||||
|
||||
The logging of the extraction plugin is displayed in the console.
|
||||
|
||||
## Locally with Docker
|
||||
|
||||
Debugging an extraction plugin via docker is a bit trickier. Java has the advantage that remote debugging is already
|
||||
baked in.
|
||||
|
||||
Using Java Remote Debug with Docker containers requires 3 distinct steps:
|
||||
|
||||
1. Build a Docker image
|
||||
2. Run the Docker image with specific Java tool options
|
||||
3. Setting breakpoints in your code
|
||||
|
||||
### Build a Docker image
|
||||
|
||||
If the Docker image is not built, run the following command to build the Docker image:
|
||||
|
||||
```bash
|
||||
mvn package docker:build
|
||||
```
|
||||
|
||||
### Run the Docker image with specific Java tool options
|
||||
|
||||
In Java, the remote debug functionality is not enabled by default. To enable the remote debug functionality, the
|
||||
following environments variable must be set in the Docker container:
|
||||
|
||||
```bash
|
||||
JAVA_TOOL_OPTIONS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"
|
||||
```
|
||||
|
||||
This environment variable allows the debugger to connect to the debuggee (application being debugged). To start the
|
||||
Docker image with the `JAVA_TOOL_OPTIONS` environment variable, the following command can be used:
|
||||
|
||||
```bash
|
||||
docker run -p 5005:5005 -e JAVA_TOOL_OPTIONS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005" your_extraction_plugin_name
|
||||
```
|
||||
|
||||
The next step is to attach the debugger to the debuggee.
|
||||
For Intellij, the instructions are clearly described on the following
|
||||
page: [Tutorial: Remote debug](https://www.jetbrains.com/help/idea/tutorial-remote-debug.html#49be7f04)
|
||||
|
||||
### Setting breakpoints in the code
|
||||
|
||||
The last step is to add breakpoints in the code.
|
||||
|
||||
### Logging in Docker
|
||||
|
||||
The logging of the extraction plugin is displayed in the console after running the `docker run` command. In addition,
|
||||
the logging is also displayed in the IntelliJ console while debugging.
|
||||
|
||||
## Kubernetes
|
||||
|
||||
In kubernetes it is currently _not_ possible to debug via Java Remote Debug because:
|
||||
|
||||
* no debug ports are published;
|
||||
* the container was not started with the environment variable `JAVA_TOOL_OPTIONS` so debugging is not enabled.
|
||||
|
||||
### Logging in Kubernetes
|
||||
|
||||
If there is authorization to the kubernetes cluster, the logging can be viewed with the following command:
|
||||
|
||||
```bash
|
||||
kubectl logs -f hansken-extraction-plugins/your_extraction_plugin_pod
|
||||
```
|
||||
|
||||
## Debug HQL
|
||||
|
||||
An HQL query can be debugged by overriding the `isVerboseLoggingEnabled()` method of the `ExtractionPluginFlits` class.
|
||||
The example below shows an example of an embedded FLITS test with verbose logging enabled.
|
||||
|
||||
```java
|
||||
public class TestPluginFlitsIT extends EmbeddedExtractionPluginFlits {
|
||||
|
||||
@Override
|
||||
public Path testPath() {
|
||||
return srcPath("integration/inputs/plugin");
|
||||
}
|
||||
|
||||
@Override
|
||||
public Path resultPath() {
|
||||
return srcPath("integration/results/embedded/plugin");
|
||||
}
|
||||
|
||||
@Override
|
||||
protected ExtractionPlugin pluginToTest() {
|
||||
return new TestPlugin();
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean regenerate() {
|
||||
return true;
|
||||
}
|
||||
|
||||
@Override
|
||||
protected boolean isVerboseLoggingEnabled() {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The following output will then be displayed in the console:
|
||||
|
||||
```text
|
||||
HQL match found for:
|
||||
$data.type=jpg
|
||||
With trace:
|
||||
dataType=jpg
|
||||
types={file, data}
|
||||
properties={data.raw.mimeType=image/jpg, path=/test-input-trace, file.name=image.jpg, name=test-input-trace, id=0}
|
||||
```
|
||||
|
||||
If the HQL query contains an error, it will be shown in the generated test results. An example of an invalid query
|
||||
is ``$data.mimeType=image/jpg`` (slash not escaped). This query will produce an error like the one shown below.
|
||||
|
||||
```json
|
||||
{
|
||||
"class": "org.hansken.plugin.extraction.hql_lite.lang.ParseException",
|
||||
"message": "HqlLiteHumanQueryParser: line 1:20 token recognition error at: '/jpg'"
|
||||
}
|
||||
```
|
||||
|
||||
.. note:: The error is only shown in the generated trace, so to find out the `ParseException` override
|
||||
the `regenerate()` method from `Flits` and then let this method return `true`.
|
||||
@@ -0,0 +1,3 @@
|
||||
# Javadoc
|
||||
|
||||
Visit the `Javadoc of the Extraction Plugins SDK API <../../_static/javadoc/index.html>`_.
|
||||
@@ -0,0 +1,42 @@
|
||||
# Packaging
|
||||
|
||||
Extraction plugins are packaged as OCI images (also known as Docker images).
|
||||
The OCI images are labeled with the PluginInfo.
|
||||
To automate packaging of a Java plugin and labeling the OCI image, the Extraction Plugin SuperPom has been configured to automate this for you.
|
||||
|
||||
If your project uses the Extraction Plugin SuperPom (see [Prerequisites](prerequisites.md)), Packaging an extraction plugin is handled by Maven.
|
||||
To package your plugin into a container image, the following command can be used:
|
||||
|
||||
```bash
|
||||
mvn package docker:build
|
||||
```
|
||||
|
||||
This will generate a plugin image:
|
||||
|
||||
* The extraction plugin is added to your local image registry
|
||||
(`docker images`),
|
||||
* The image name is `extraction-plugin/PLUGINID`, e.g.
|
||||
`extraction-plugin/nfi.nl/extract/chat/whatsapp`,
|
||||
* The image is labeled with two tags: `latest`, and your plugin version.
|
||||
|
||||
It is possible to apply extra arguments to the docker command [as described here](http://dmp.fabric8.io/#docker:build).
|
||||
For example, to specify a proxy, use the following command:
|
||||
|
||||
```bash
|
||||
mvn package docker:build -Ddocker.buildArg.https_proxy=https://proxy:8001
|
||||
```
|
||||
|
||||
Once your plugin is packaged, it can be published or 'uploaded' to Hansken.
|
||||
See ":ref:`upload_plugin`" for instructions.
|
||||
|
||||
.. _java_superpom_podman:
|
||||
|
||||
Note: if your build environment does not have Docker available, you can use
|
||||
[podman](https://podman.io/) as an alternative. Install podman on your machine
|
||||
or build agent, and run the following commands _before_ invoking the
|
||||
`mvn package docker:build` command:
|
||||
|
||||
```bash
|
||||
podman system service --time=0 unix:/run/user/$(id -u)/podman/podman.sock &
|
||||
export DOCKER_HOST="unix:/run/user/$(id -u)/podman/podman.sock"
|
||||
```
|
||||
@@ -0,0 +1,41 @@
|
||||
# Prerequisites
|
||||
|
||||
Required software:
|
||||
|
||||
* Java 11 or higher
|
||||
* Docker for [packaging](packaging.md) and publishing plugins
|
||||
(or use a Docker alternative such as `podman`)
|
||||
* Maven (recommended, build automation tool)
|
||||
|
||||
Required dependencies:
|
||||
* All required project dependencies to build extraction plugins are published on the public Maven Central, under `org.hansken.plugin.extraction:plugin-super-pom`.
|
||||
For maven based extraction plugins, the following `pom.xml` snippet can be used as basis of a plugin:
|
||||
```xml
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
|
||||
<parent>
|
||||
<groupId>org.hansken.plugin.extraction</groupId>
|
||||
<artifactId>plugin-super-pom</artifactId>
|
||||
<version>SET_THE_SDK_VERSION_HERE</version>
|
||||
</parent>
|
||||
|
||||
<artifactId>CHOOSE_YOUR_ARTIFACTID_HERE</artifactId>
|
||||
<version>SET_THE_PLUGIN_VERSION_HERE</version>
|
||||
|
||||
<licenses>
|
||||
<license>
|
||||
<name>The Apache Software License, Version 2.0</name>
|
||||
<url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
|
||||
<distribution>repo</distribution>
|
||||
</license>
|
||||
</licenses>
|
||||
|
||||
<properties>
|
||||
<mainClass>SET_THE_PLUGIN_MAIN_CLASS_HERE</mainClass>
|
||||
</properties>
|
||||
</project>
|
||||
```
|
||||
@@ -0,0 +1,246 @@
|
||||
# Java code snippets
|
||||
|
||||
This page contains Java code snippets for common patterns that will be used when writing a plugin.
|
||||
|
||||
## RandomAccessData as InputStream
|
||||
|
||||
In Java, `InputStream` is a common type to pass data to another class or method. The SDK provides a simple utility to
|
||||
use a `RandomAccessData` as `InputStream`.
|
||||
|
||||
Add the following import to your code:
|
||||
|
||||
```java
|
||||
import org.hansken.plugin.extraction.core.data.RandomAccessDatas;
|
||||
```
|
||||
|
||||
Next we can create an `InputStream` from the `RandomAccessData` as shown in the following snippet. Note that the
|
||||
`InputStream` is created using a `try-with-resources`-statement. This ensures that the `InputStream` is correctly closed
|
||||
when the `InputStream` is no longer required.
|
||||
|
||||
```java
|
||||
RandomAccessData traceData=...;
|
||||
try(InputStream asInputStream=RandomAccessDatas.asInputStream(traceData)){
|
||||
// use the InputStream here
|
||||
}
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
* the created `InputStream` is *not* thread-safe,
|
||||
* the created `InputStream` changes state in the provided `RandomAccessData`
|
||||
(e.g. when data is read, the position updated in both the `InputStream` *and*
|
||||
the `RandomAccessData` instances),
|
||||
* for more details on the implementation of the `InputStream`, refer to the `RandomAccessDataInputStream` JavaDoc.
|
||||
|
||||
|
||||
.. _tracelets java:
|
||||
|
||||
## Adding tracelets
|
||||
|
||||
In the following Java example, a "classification" :ref:`tracelet<tracelets>` is added to a trace. The tracelet consists
|
||||
of a list of four properties, namely "class", "confidence", "modelName" and "modelVersion".
|
||||
|
||||
```java
|
||||
trace.addTracelet("prediction", tracelet -> tracelet
|
||||
.set("type", "classification")
|
||||
.set("class", "telephone")
|
||||
.set("label", "label")
|
||||
.set("confidence", 0.8f)
|
||||
.set("embedding", Vector.of(1,2,3))
|
||||
.set("modelName", "yolo")
|
||||
.set("modelVersion", "2.0"));
|
||||
```
|
||||
or
|
||||
```java
|
||||
trace.addTracelet(new Tracelet("prediction", List.of(
|
||||
new TraceletProperty("prediction.type","classification"),
|
||||
new TraceletProperty("prediction.class","telephone"),
|
||||
new TraceletProperty("prediction.label","label"),
|
||||
new TraceletProperty("prediction.confidence",0.8f))),
|
||||
new TraceletProperty("prediction.embedding", Vector.of(1,2,3)),
|
||||
new TraceletProperty("prediction.modelName","yolo"),
|
||||
new TraceletProperty("prediction.modelVersion","2.0"));
|
||||
```
|
||||
|
||||
|
||||
.. _datastreams java:
|
||||
|
||||
## Adding data to a trace
|
||||
|
||||
Traces can have data attatched to them. See :ref:`datastreams` for more information.
|
||||
The following two snippets demonstrate how to add data to a trace.
|
||||
|
||||
It is currently not possible to verify that a specific data stream is already set or not.
|
||||
|
||||
### Data Transformations
|
||||
|
||||
The most efficient way to add data to a trace is using data transformations.
|
||||
See :doc:`../concepts/data_transformations` for more details.
|
||||
|
||||
The following example sets a new data stream with dataType `html` on a trace, by setting a ranged data transformation:
|
||||
|
||||
```java
|
||||
trace.setData("html", RangedDataTransformation.builder().addRange(offset, length).build());
|
||||
```
|
||||
|
||||
The following example creates a child trace and sets a new datastream with dataType `raw` on it, by setting a ranged
|
||||
data transformation with two ranges:
|
||||
|
||||
```java
|
||||
trace.newChild(format("lineNumber %d", lineNumber), child -> {
|
||||
child.setData("raw", RangedDataTransformation.builder()
|
||||
.addRange(10, 20)
|
||||
.addRange(50, 30)
|
||||
.build());
|
||||
});
|
||||
```
|
||||
|
||||
### Blobs
|
||||
|
||||
It is not always possible to create a transormation for the data that has to be
|
||||
added to a trace. For example the data is a result of a computation, and not
|
||||
a direct subset of another data stream..
|
||||
|
||||
The following examples show how to creates a new data stream of dataType `raw` on a trace.
|
||||
|
||||
In case all data is stored in a `byte[]`, we can add the byte array to the data stream with:
|
||||
|
||||
```java
|
||||
final byte[] rawBytes = {.....};
|
||||
trace.setData("raw", writer -> writer.write(rawBytes));
|
||||
```
|
||||
|
||||
Alternatively, if the data is available in an `InputStream` the data can be added with:
|
||||
|
||||
```java
|
||||
final InputStream inputStream = ...;
|
||||
trace.setData("raw", inputStream);
|
||||
```
|
||||
|
||||
## Specifying system resources
|
||||
|
||||
In the `PluginInfo` you can specify **maximum** system resource metrics for a plugin. These are used for scaling the
|
||||
number of pods as described [here](../concepts/kubernetes_autoscaling.md#Autoscaling). To run a plugin with 0.5 cpu (=
|
||||
0.5 vCPU/Core/hyperthread) and 1 gb memory, for example, the following configuration can be added to `PluginInfo`:
|
||||
|
||||
```java
|
||||
PluginInfo.builderFor(this)
|
||||
...
|
||||
.pluginResources(PluginResources.builder()
|
||||
.maximumCpu(0.5f)
|
||||
.maximumMemory(1000)
|
||||
.build())
|
||||
.build();
|
||||
```
|
||||
|
||||
.. _java_snippets_deferred:
|
||||
|
||||
## Deferred Extraction Plugins
|
||||
|
||||
Using a deferred plugin requires inheriting the `DeferredExtractionPlugin` base class. This allows access to
|
||||
a ``TraceSearcher`` object in the process function to search for traces.
|
||||
|
||||
```java
|
||||
public class ExampleDeferred extends DeferredExtractionPlugin {
|
||||
@Override
|
||||
public PluginInfo pluginInfo();
|
||||
|
||||
@Override
|
||||
public void process(final Trace trace, final ExtractionContext context,
|
||||
final TraceSearcher searcher) {
|
||||
final SearchResult result = searcher.search("file.extension=asc", 10);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The ``search`` method accepts a HQL query and a count, which represents the maximum number of traces to return. It may
|
||||
be useful to specifically search for traces from the image being extracted. Add ``"image:" + trace.get("image")`` to
|
||||
your query. The query of the provided example could be extended like this:
|
||||
`"file.extension = asc AND image:" + trace.get("image")`.
|
||||
|
||||
The traces contained in the ``SearchResult`` are returned as a stream.
|
||||
|
||||
```java
|
||||
final Stream<Trace> stream = result.getTraces();
|
||||
stream.limit(5);
|
||||
```
|
||||
|
||||
|
||||
## Logging
|
||||
|
||||
The logging is provided by Log4j 2 with a SLF4J binding. The Log4j 2 SLF4J binding allows applications coded to the
|
||||
SLF4J API to use Log4j 2 as the implementation.
|
||||
|
||||
### Usage
|
||||
|
||||
Here is an example illustrating how to log something with SLF4J. It begins by getting a logger with the name "LOG". This
|
||||
logger is in turn used to log the message `I'm logging a variable 1234!`.
|
||||
|
||||
```java
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
public class Example {
|
||||
private static final Logger LOG = LoggerFactory.getLogger(Example.class);
|
||||
|
||||
public void example() {
|
||||
final int aNumber = 1234;
|
||||
// logs to console: I'm logging a variable 1234!
|
||||
LOG.info("I'm logging a variable {}!", aNumber);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Customize logging
|
||||
|
||||
It's easy to change the logging format with a file called `log4j2.xml`. If desired, this file must be in the `resources`
|
||||
folder, for example `src/main/resources/log4j2.xml`
|
||||
|
||||
```xml
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<configuration>
|
||||
<appenders>
|
||||
<console name="stdout" target="SYSTEM_OUT">
|
||||
<patternLayout
|
||||
pattern="%-5p|%d{yyyy-MM-dd HH:mm:ss}|%-20.20t|%-32.32c{1}|%m%n"/>
|
||||
</console>
|
||||
</appenders>
|
||||
<loggers>
|
||||
<root level="info">
|
||||
<appenderRef ref="stdout"/>
|
||||
</root>
|
||||
</loggers>
|
||||
</configuration>
|
||||
```
|
||||
|
||||
.. warning:: Be careful with logging sensitive information.
|
||||
|
||||
.. note:: More information about customizing the logging can be found `here <https://logging.apache.org/log4j/2.x>`_.
|
||||
|
||||
.. note:: The default logger is pre-configured to log `INFO` to `STDOUT` (see the configuration above)
|
||||
|
||||
.. note:: Log4j 2 supports various logging formats, including xml, yaml, json, properties, etc.
|
||||
Currently, only the xml format is supported.
|
||||
|
||||
.. note:: Contact your Hansken administrator for more information on where to find logs for your Hansken environment.
|
||||
|
||||
## [EXPERIMENTAL FEATURE] Adding previews to a trace
|
||||
|
||||
.. warning:: This is an experimental feature, which might change or get removed in future releases.
|
||||
|
||||
Example:
|
||||
|
||||
```java
|
||||
public class ExamplePlugin extends ExtractionPlugin {
|
||||
@Override
|
||||
public PluginInfo pluginInfo();
|
||||
|
||||
@Override
|
||||
public void process(final Trace trace, final DataContext context) {
|
||||
final byte[] previewData;
|
||||
// set the preview data for the image/png MIME-type
|
||||
trace.set("preview.image/png", previewData);
|
||||
trace.set("preview.image/png", previewData);
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,156 @@
|
||||
# Using the Test Framework in Java
|
||||
|
||||
.. _java testing:
|
||||
|
||||
This section assumes you use the same setup as is used in the
|
||||
[Extraction Plugin Examples](https://git.eminjenv.nl/hanskaton/hansken-extraction-plugin-sdk/examples).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Java Plugins can use the `plugin-super-pom` as maven parent, which makes sure the
|
||||
FLITS [Test Framework](../concepts/test_framework.md) is included in the build.
|
||||
|
||||
```xml
|
||||
|
||||
<parent>
|
||||
<groupId>org.hansken.plugin.extraction</groupId>
|
||||
<artifactId>plugin-super-pom</artifactId>
|
||||
<version>0.4.3</version>
|
||||
</parent>
|
||||
```
|
||||
|
||||
## Embedded Testing versus Remote Testing
|
||||
|
||||
There are ways of integration testing a plugin with the Test Framework:
|
||||
|
||||
* **Embedded testing**: Here the plugin is run directly from a JUnit test without using the gRPC layer.
|
||||
* **Remote testing**: Here the test will start an ExtractionPluginServer that will serve the plugin. All communication
|
||||
is between server and plugin is done using gRPC.
|
||||
|
||||
See below for an example of each way of testing.
|
||||
The [Extraction Plugin Examples](https://git.eminjenv.nl/hanskaton/hansken-extraction-plugin-sdk/examples) contains many
|
||||
more examples.
|
||||
|
||||
### Embedded Testing example
|
||||
|
||||
Embedded tests can be run as a unit test.
|
||||
|
||||
```java
|
||||
import static nl.minvenj.nfi.flits.util.FlitsUtil.srcPath;
|
||||
|
||||
import java.nio.file.Path;
|
||||
|
||||
import org.hansken.plugin.extraction.api.ExtractionPlugin;
|
||||
import org.hansken.plugin.extraction.test.EmbeddedExtractionPluginFlits;
|
||||
|
||||
/**
|
||||
* An integration test for MyPlugin.
|
||||
*/
|
||||
class MyPluginIT extends EmbeddedExtractionPluginFlits {
|
||||
|
||||
@Override
|
||||
protected ExtractionPlugin pluginToTest() {
|
||||
// MyPlugin is a class implementing the ExtractionPlugin interface,
|
||||
// with pluginInfo() and process() methods.
|
||||
return new MyPlugin();
|
||||
}
|
||||
|
||||
@Override
|
||||
public Path testPath() {
|
||||
// Provide the folder containing input files. For examples, see
|
||||
// https://git.eminjenv.nl/hanskaton/hansken-extraction-plugin-sdk/examples.
|
||||
return srcPath("integration/inputs");
|
||||
}
|
||||
|
||||
@Override
|
||||
public Path resultPath() {
|
||||
// Provide the folder containing result files. For examples, see
|
||||
// https://git.eminjenv.nl/hanskaton/hansken-extraction-plugin-sdk/examples.
|
||||
return srcPath("integration/results");
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean regenerate() {
|
||||
// Returning false means the test will fail if the result files don't
|
||||
// match the outcome of the=++ test. Returning true means the test create
|
||||
// new result files .
|
||||
return false;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Remote Testing example
|
||||
|
||||
Note that the following example serves the plugin by using `ExtractionServer`.
|
||||
|
||||
```java
|
||||
import static nl.minvenj.nfi.flits.util.FlitsUtil.srcPath;
|
||||
|
||||
import java.nio.file.Path;
|
||||
|
||||
import org.hansken.plugin.extraction.runtime.grpc.client.ExtractionPluginClient;
|
||||
import org.hansken.plugin.extraction.runtime.grpc.server.ExtractionPluginServer;
|
||||
import org.hansken.plugin.extraction.test.plugins.DataTransformationsPlugin;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
|
||||
public class RemoteTransformationPluginFlitsIT extends RemoteExtractionPluginFlits {
|
||||
|
||||
private static ExtractionPluginServer _server;
|
||||
private static ExtractionPluginClient _client;
|
||||
|
||||
@BeforeAll
|
||||
public static void init() throws Exception {
|
||||
final int port = 8999;
|
||||
|
||||
// Serve MyPlugin.
|
||||
// MyPlugin is a class implementing the ExtractionPlugin interface, with PluginInfo and Process methods.
|
||||
_server = ExtractionPluginServer.serve(port, MyPlugin::new);
|
||||
|
||||
// Create an ExtractionPluginClient
|
||||
_client = new ExtractionPluginClient("localhost", _server.getListeningPort());
|
||||
}
|
||||
|
||||
@AfterAll
|
||||
public static void destruct() {
|
||||
// At the end of the test, make sure the server and client are closed.
|
||||
if (_server != null) {
|
||||
_server.close();
|
||||
}
|
||||
if (_client != null) {
|
||||
_client.close();
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public Path testPath() {
|
||||
// Provide the folder containing input files. For examples, see https://git.eminjenv.nl/hanskaton/hansken-extraction-plugin-sdk/examples.
|
||||
return srcPath("integration/inputs");
|
||||
}
|
||||
|
||||
@Override
|
||||
public Path resultPath() {
|
||||
// Provide the folder containing result files. For examples, see https://git.eminjenv.nl/hanskaton/hansken-extraction-plugin-sdk/examples.
|
||||
return srcPath("integration/results");
|
||||
}
|
||||
|
||||
@Override
|
||||
protected ExtractionPluginClient pluginToTest() {
|
||||
// For Remote testing, the test won't talk directly to the plugin, but to the client.
|
||||
// The client will use gRPC to communicate with the served plugin.
|
||||
return _client;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean regenerate() {
|
||||
// Returning false means the test will fail if the result files don't match the outcome of the test.
|
||||
// Returning true means the test create new result files.
|
||||
return false;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
.. note:: Note that with a `RemoteTransformationPluginFlitsIT` it is possible to start a docker image of a plugin and
|
||||
run remote tests against it using your own testdata. To do this, simply remove all `_server` code and manually start
|
||||
your plugin in a docker container. Then run the test against the docker container by setting the correct url and
|
||||
port, presumably `new ExtractionPluginClient("localhost", 8999)`.
|
||||
Reference in New Issue
Block a user