Payload Mapping

You use payload mapping to transform messages between your device’s native format and Ditto Protocol JSON.

Overview

Devices rarely send data in Ditto Protocol format. A device might send:

{"val": "23.42 °C", "ts": 1523946112727}

Or even binary:

0x08BD

Payload mapping bridges this gap by converting between device-native formats and the structured Ditto Protocol that Ditto requires.

Built-in mappers

Ditto mapper

The default mapper. Assumes messages are in Ditto Protocol JSON. No configuration required – use the alias Ditto directly.

JavaScript mapper

Transforms arbitrary payloads using custom JavaScript scripts executed in a sandboxed Rhino engine. See the JavaScript mapping engine section for details.

Options:

Normalized mapper

Transforms created, modified, and deleted events to a normalized JSON structure. Other message types are dropped.

Options:

Example input:

{
  "topic": "thing/id/things/twin/events/modified",
  "path": "/features/sensors/properties/temperature/value",
  "value": 42
}

Normalized output:

{
  "thingId": "thing:id",
  "features": {
    "sensors": {
      "properties": {
        "temperature": { "value": 42 }
      }
    }
  },
  "_context": {
    "topic": "thing/id/things/twin/events/modified",
    "path": "/features/sensors/properties/temperature/value",
    "value": 42
  }
}

For deleted events, the mapper includes a _deleted field with the deletion timestamp:

{
  "topic": "thing/id/things/twin/events/deleted",
  "headers": { "content-type": "application/json" },
  "path": "/",
  "value": null
}

Normalized output:

{
  "thingId": "thing:id",
  "_deleted": "2023-12-01T10:30:00Z",
  "_context": {
    "topic": "thing/id/things/twin/events/deleted",
    "path": "/",
    "value": null,
    "headers": {
      "content-type": "application/json"
    }
  }
}

The _deleted field contains the ISO-8601 timestamp when the thing was deleted. This field is only added for complete thing deletions.

When includeDeletedFields is enabled, partial deletions and merge-patch deletions are tracked in _deletedFields. The _deletedFields object mirrors the JSON structure of the deleted paths and stores ISO-8601 timestamps at the leaf nodes:

{
  "thingId": "thing:id",
  "_deletedFields": {
    "attributes": {
      "location": {
        "mountedOn": "2025-01-27T10:00:00Z"
      }
    },
    "features": {
      "myFeature": "2025-01-27T11:00:00Z"
    }
  }
}

ConnectionStatus mapper

Transforms ttd and creation-time headers (from Eclipse Hono device notifications) into a ModifyFeature command that updates a ConnectionStatus feature.

Typically used alongside another mapper: "payloadMapping": ["Ditto", "connectionStatus"]

Options:

Example of a resulting ConnectionStatus feature:

{
  "thingId": "eclipse:ditto",
  "features": {
    "ConnectionStatus": {
      "definition": [ "org.eclipse.ditto:ConnectionStatus:1.0.0" ],
      "properties": {
        "status": {
          "readySince": "2019-10-29T14:16:18Z",
          "readyUntil": "2019-10-29T14:21:18Z"
        }
      }
    }
  }
}

Use the ConnectionStatus mapper alongside another mapper in a source configuration:

{
  "addresses": ["<source>"],
  "authorizationContext": ["ditto:inbound"],
  "payloadMapping": ["Ditto", "connectionStatus"]
}

RawMessage mapper

Maps message command/response payloads directly to/from the external message format. The encoding is determined by the content type.

For incoming messages, the mapper wraps the payload in a message command envelope. For outgoing messages, the mapper extracts the "value" field for publishing.

Messages with content-type: application/vnd.eclipse.ditto+json fall through to the Ditto mapper.

Options:

Key incoming headers (all support placeholders):

The mapper maps between a feature message command response like:

{
  "topic": "org.eclipse.ditto/smartcoffee/things/live/messages/heatUp",
  "headers": { "content-type": "application/octet-stream" },
  "path": "/features/water-tank/inbox/messages/heatUp",
  "value": "AQIDBAUG",
  "status": 200
}

and an AMQP, MQTT 5, or Kafka message with payload of 6 bytes:

0x01 02 03 04 05 06

with headers:

content-type: application/octet-stream
status: 200
ditto-message-subject: heatUp
ditto-message-direction: TO
ditto-message-thing-id: org.eclipse.ditto:smartcoffee
ditto-message-feature-id: water-tank

Headers are lost for connection protocols without application headers such as MQTT 3.

Example configuration:

{
  "outgoingContentType": "application/octet-stream",
  "incomingMessageHeaders": {
    "content-type": "{{ header:content-type | fn:default('application/octet-stream') }}",
    "status": "{{ header:status }}",
    "ditto-message-subject": "{{ header:ditto-message-subject | fn:default('fallback-subject') }}",
    "ditto-message-direction": "TO",
    "ditto-message-thing-id": "{{ header:ditto-message-thing-id | fn:default('ns:fallback-thing') }}",
    "ditto-message-feature-id": "{{ header:ditto-message-feature-id }}"
  }
}

ImplicitThingCreation mapper

Automatically creates a thing when an incoming message arrives. The thing structure is defined in the thing option as a JSON template with placeholder support.

Options:

• thing (required): The values of the thing that is created implicitly. It can contain fixed values or header placeholders (e.g. {{ header:device_id }}).

  • The following placeholders may be used inside the "thing" JSON:

    Placeholder	Description
    {{ header:<header-name> }}	Header value from the external message, e.g. from protocol headers
    {{ request:subjectId }}	The first authenticated subjectId which did the request
    {{ time:now }}	The current timestamp in ISO-8601 format as string in UTC timezone
    {{ time:now_epoch_millis }}	The current timestamp in “milliseconds since epoch” formatted as string

  • The "thing" JSON may also include:

    • an inline policy: "_policy" containing the Policy JSON
    • a “copy policy from” statement: "_copyPolicyFrom" - see also create Thing alternatives
      • either including a policyId to copy from
      • or containing the link to a thing to copy the policy from in the form: {{ ref:things/<theThingId>/policyId }}
• commandHeaders (optional, default: {"If-None-Match": "*"}): The Ditto headers to use for constructing the “create thing” command.

  • The following placeholders may be used:

    Placeholder	Description
    {{ header:<header-name> }}	Header value from the external message, e.g. from protocol headers
    {{ request:subjectId }}	The first authenticated subjectId which did the request
    {{ time:now }}	The current timestamp in ISO-8601 format as string in UTC timezone
    {{ time:now_epoch_millis }}	The current timestamp in “milliseconds since epoch” formatted as string

• allowPolicyLockout (optional, default: true): Whether it should be allowed to create policies without having WRITE permissions in the created policy for the subject which creates the policy.

{
  "thing": {
    "thingId": "{{ header:device_id }}",
    "attributes": {
      "CreatedBy": "ImplicitThingCreation"
    }
  },
  "commandHeaders": {
    "If-None-Match": "*",
    "correlation-id": "{{ header:correlation-id }}"
  },
  "allowPolicyLockout": true
}

UpdateTwinWithLiveResponse mapper

Creates a merge Thing command from a live retrieve response, patching the live data into the twin.

Options:

• dittoHeadersForMerge (optional): The Ditto headers to use for constructing the “merge thing” command, may for example add a condition to apply in order to update the twin (default: "response-required": false, "if-match": "*").

  • The following placeholders may be used:

    Placeholder	Description
    {{ header:<header-name> }}	Header value from the external message, e.g. from protocol headers
    {{ request:subjectId }}	The first authenticated subjectId which did the request
    {{ time:now }}	The current timestamp in ISO-8601 format as string in UTC timezone
    {{ time:now_epoch_millis }}	The current timestamp in “milliseconds since epoch” formatted as string

Example configuration:

{
  "dittoHeadersForMerge": {
    "if-match": "*",
    "response-required": false,
    "put-metadata": [
      {"key":"*/updated-by","value":"{{ request:subjectId }}"},
      {"key":"*/updated-via","value":"device-live-response"},
      {"key":"*/update-hint","value":"{{ header:some-custom-hint }}"},
      {"key":"*/updated-at","value":"{{ time:now }}"}
    ]
  }
}

CloudEvents mapper

Maps incoming CloudEvents to Ditto Protocol. Supports both Binary and Structured CloudEvents.

Note: The mapper supports incoming Structured CloudEvents messages with content-type:application/cloudevents+json and Binary CloudEvents messages with content-type:application/vnd.eclipse.ditto+json.

CloudEvents examples

Incoming messages need to have the mandatory CloudEvents fields. For example, a Binary CloudEvent for Ditto would look like this:

  headers:
      ce-specversion:1.0
      ce-id:some-id
      ce-type:some-type
      ce-source:generic-producer
      content-type:application/vnd.eclipse.ditto+json

{
  "topic": "my.sensors/sensor01/things/twin/commands/modify",
  "path": "/",
  "value": {
    "thingId": "my.sensors:sensor01",
    "policyId": "my.test:policy",
    "attributes": {
      "manufacturer": "Well known sensors producer",
      "serial number": "100",
      "location": "Ground floor"
    },
    "features": {
      "measurements": {
        "properties": {
          "temperature": 100,
          "humidity": 0
        }
      }
    }
  }
}

A Structured CloudEvent for Ditto would look like this:

headers:
  content-type:application/cloudevents+json

{
  "specversion": "1.0",
  "id": "3212e",
  "source": "http:somesite.com",
  "type": "com.site.com",
  "data": {
    "topic": "my.sensors/sensor01/things/twin/commands/modify",
    "path": "/",
    "value": {
      "thingId": "my.sensors:sensor01",
      "policyId": "my.test:policy",
      "attributes": {
        "manufacturer": "Well known sensors producer",
        "serial number": "100",
        "location": "Ground floor"
      },
      "features": {
        "measurements": {
          "properties": {
            "temperature": 100,
            "humidity": 0
          }
        }
      }
    }
  }
}

Using multiple mappers

Reference multiple mappers in a source’s payloadMapping array. Define custom mappers in mappingDefinitions:

{
  "name": "exampleConnection",
  "sources": [{
    "addresses": ["<source>"],
    "authorizationContext": ["ditto:inbound"],
    "payloadMapping": ["Ditto", "status"]
  }],
  "mappingDefinitions": {
    "status": {
      "mappingEngine": "ConnectionStatus",
      "options": {
        "thingId": "{{ header:device_id }}"
      }
    }
  }
}

Mapping conditions

You can add incomingConditions and outgoingConditions to control when a mapper executes. All conditions must evaluate to true for the mapping to run:

{
  "mappingDefinitions": {
    "status": {
      "mappingEngine": "ConnectionStatus",
      "incomingConditions": {
        "sampleCondition": "fn:filter(header:incoming-mapping-required,'eq','true')"
      },
      "outgoingConditions": {
        "sampleCondition": "fn:filter(header:outgoing-mapping-required,'eq','true')"
      },
      "options": {
        "thingId": "{{ header:device_id }}"
      }
    }
  }
}

JavaScript mapping engine

Ditto uses the Rhino JavaScript engine (version 1.7.14, ES6 flag enabled) with strict sandboxing for security.

Sandboxing constraints

• No access to Java packages or classes
• No file access, network calls, or exit/quit/print
• Endless loops and deep recursion are terminated
• Script file size is limited
• No foreign JS library loading (unless included inline)

Check Rhino compatibility for supported ES6 features.

Helper libraries

You can load these libraries via specificConfig options:

• bytebuffer.js – ArrayBuffer manipulation
• long.js – 64-bit integer support

Adding CommonJS modules

Configure CONNECTIVITY_MESSAGE_MAPPING_JS_COMMON_JS_MODULE_PATH to point to a directory containing CommonJS modules (for example, via a Docker volume mount):

CONNECTIVITY_MESSAGE_MAPPING_JS_COMMON_JS_MODULE_PATH=/opt/commonjs-modules/

Then use require() in your scripts:

var Pbf = require('pbf');

Helper functions

Ditto provides these functions under the Ditto scope:

// Builds a Ditto Protocol message from the passed parameters.
function buildDittoProtocolMsg(namespace, name, group, channel,
    criterion, action, path, dittoHeaders, value, status, extra) {
    const topic = buildTopic(namespace, name, group, channel, criterion, action);

    return {
        topic: topic,
        path: path,
        headers: dittoHeaders,
        value: value,
        status: status,
        extra: extra,
    };
}

// Builds a Ditto Protocol topic string.
function buildTopic(namespace, name, group, channel, criterion, action) {
    const topicChannel = 'none' === channel ? '' : '/' + channel;

    return namespace + "/" + name + "/" + group + topicChannel + "/" + criterion + "/" + action;
}

// Builds an external message from the passed parameters.
function buildExternalMsg(headers, textPayload, bytePayload, contentType) {

  return {
    headers: headers,
    textPayload: textPayload,
    bytePayload: bytePayload,
    contentType: contentType,
  };
}

// Transforms an ArrayBuffer to a String (unsigned 8-bit integers).
function arrayBufferToString(arrayBuffer) {

  return String.fromCharCode.apply(null, new Uint8Array(arrayBuffer));
}

// Transforms a String to an ArrayBuffer (unsigned 8-bit integers).
function stringToArrayBuffer(string) {

  let buf = new ArrayBuffer(string.length);
  let bufView = new Uint8Array(buf);
  for (let i=0, strLen=string.length; i<strLen; i++) {
    bufView[i] = string.charCodeAt(i);
  }
  return buf;
}

// Transforms an ArrayBuffer to a ByteBuffer (requires bytebuffer.js).
function asByteBuffer(arrayBuffer) {

  let byteBuffer = new ArrayBuffer(arrayBuffer.byteLength);
  new Uint8Array(byteBuffer).set(new Uint8Array(arrayBuffer));
  return dcodeIO.ByteBuffer.wrap(byteBuffer);
}

Mapping incoming messages

Implement mapToDittoProtocolMsg to convert external payloads to Ditto Protocol:

function mapToDittoProtocolMsg(
  headers,
  textPayload,
  bytePayload,
  contentType
) {

  // Insert/adapt your mapping logic here.
  // Use Ditto.buildDittoProtocolMsg to build Ditto Protocol messages from incoming payload.
  if (contentType === 'application/vnd.eclipse.ditto+json') {
    // Message is already in Ditto Protocol format -- use directly
    return JSON.parse(textPayload);
  } else if (contentType === 'application/octet-stream') {
    // Binary payload -- assume Ditto Protocol message (JSON)
    try {
      return JSON.parse(Ditto.arrayBufferToString(bytePayload));
    } catch (e) {
      // parsing failed (no JSON document); drop the message
      return null;
    }
  } else if (contentType === 'application/json') {
    let parsedJson = JSON.parse(textPayload);
    value = parsedJson.number1 + parsedJson['sub-field']; // access JSON keys with dashes using bracket notation
    // determine these variables from parsedJson and headers:
    let namespace = "";
    let name = "";
    let group = "things";
    let channel = "twin";
    let criterion = "commands";
    let action = "modify";
    let path = "/attributes";
    let dittoHeaders = {};
    let value = {
      "a": 1
    };
    return Ditto.buildDittoProtocolMsg(
      namespace,
      name,
      group,
      channel,
      criterion,
      action,
      path,
      dittoHeaders,
      value)
  }
  // no mapping logic matched; drop the message
  return null;
}

Return a single Ditto Protocol message, an array of messages, or null to drop the message.

For full access to the external message object, implement mapToDittoProtocolMsgWrapper instead:

function mapToDittoProtocolMsgWrapper(externalMsg) {
  let headers = externalMsg.headers;
  let textPayload = externalMsg.textPayload;
  let bytePayload = externalMsg.bytePayload;
  let contentType = externalMsg.contentType;
  return mapToDittoProtocolMsg(headers, textPayload, bytePayload, contentType);
}

Mapping outgoing messages

Implement mapFromDittoProtocolMsg to convert Ditto Protocol messages to external format:

function mapFromDittoProtocolMsg(namespace, name, group, channel,
    criterion, action, path, dittoHeaders, value, status, extra) {
  let headers = dittoHeaders;
  let textPayload = JSON.stringify(
    Ditto.buildDittoProtocolMsg(namespace, name, group, channel,
      criterion, action, path, dittoHeaders, value, status, extra)
  );
  return Ditto.buildExternalMsg(headers, textPayload, null,
    'application/vnd.eclipse.ditto+json');
}

For access to the full Ditto Protocol message (including revision), implement mapFromDittoProtocolMsgWrapper:

function mapFromDittoProtocolMsgWrapper(dittoProtocolMsg) {

  let topic = dittoProtocolMsg.topic;
  let splitTopic = topic.split("/");

  let namespace = splitTopic[0];
  let name = splitTopic[1];
  let group = splitTopic[2];

  let channel;
  let criterion;
  let action;
  if (hasChannel(group)) {
    channel = splitTopic[3];
    criterion = splitTopic[4];
    action = splitTopic[5];
  } else {
    channel = 'none';
    criterion = splitTopic[3];
    action = splitTopic[4];
  }

  let path = dittoProtocolMsg.path;
  let dittoHeaders = dittoProtocolMsg.headers;
  let value = dittoProtocolMsg.value;
  let status = dittoProtocolMsg.status;
  let extra = dittoProtocolMsg.extra;

  return mapFromDittoProtocolMsg(namespace, name, group, channel, criterion, action, path, dittoHeaders, value, status, extra);
}

Working with byte payloads

Use TypedArrays or DataViews to process binary data:

// TypedArray approach
let bytes = new Uint8Array(bytePayload);

// DataView approach (mixed types)
let view = new DataView(bytePayload);
view.getInt8(0);                       // 8-bit signed integer (byte) at offset 0
view.getUint16(1);                     // 16-bit unsigned integer (unsigned short) at offset 1
let temp = view.getInt16(0) / 100.0;   // 16-bit signed int at offset 0
let pressure = view.getInt16(2);       // 16-bit signed int at offset 2
let humidity = view.getUint8(4);       // 8-bit unsigned int at offset 4

Or use ByteBuffer.js (load with "loadBytebufferJS": "true"):

let byteBuf = Ditto.asByteBuffer(bytePayload);
let numberFromBytes = parseInt(byteBuf.toHex(), 16);

let base64encoded = byteBuf.toBase64();
let buf = dcodeIO.ByteBuffer.fromBase64(base64encoded);

buf.readInt(); // read a 32bit signed integer + advances the offset in the buffer
buf.readUTF8String(4); // read 4 characters of UTF-8 encoded string + advances the offset in the buffer
buf.remaining(); // gets the number of remaining readable bytes in the buffer

Check the ByteBuffer API documentation for the full list of operations.

JavaScript examples

Text payload example

Device sends JSON telemetry:

{"temp": "23.42 °C", "hum": 78, "pres": {"value": 760, "unit": "mmHg"}}

Mapping to update thing features:

function mapToDittoProtocolMsg(headers, textPayload, bytePayload, contentType) {
  if (contentType !== 'application/json') return null;

  let jsonData = JSON.parse(textPayload);
  let value = {
    temperature: { properties: { value: parseFloat(jsonData.temp.split(" ")[0]) } },
    pressure: { properties: { value: jsonData.pres.value } },
    humidity: { properties: { value: jsonData.hum } }
  };

  return Ditto.buildDittoProtocolMsg(
    'org.eclipse.ditto', headers['device_id'],
    'things', 'twin', 'commands', 'modify',
    '/features', headers, value
  );
}

Send this payload via Eclipse Hono’s MQTT adapter:

mosquitto_pub -u 'sensor1@DEFAULT_TENANT' -P hono-secret -t telemetry -m '{"temp": "23.42 °C","hum": 78,"pres": {"value": 760,"unit": "mmHg"}}'

The digital twin is updated by applying the script and extracting the relevant values from the textPayload.

Binary payload example

Device sends 5 bytes as hexadecimal 0x09EF03F72A:

• the first 2 bytes 09 EF represent the temperature as 16-bit signed integer (not a float, to save space)
• the next 2 bytes 03 F7 represent the pressure as 16-bit signed integer
• the last byte 2A represents the humidity as 8-bit unsigned integer

function mapToDittoProtocolMsg(headers, textPayload, bytePayload, contentType) {
  if (contentType !== 'application/octet-stream') return null;

  let view = new DataView(bytePayload);
  let value = {
    temperature: { properties: { value: view.getInt16(0) / 100.0 } },
    pressure: { properties: { value: view.getInt16(2) } },
    humidity: { properties: { value: view.getUint8(4) } }
  };

  return Ditto.buildDittoProtocolMsg(
    'org.eclipse.ditto', headers['device_id'],
    'things', 'twin', 'commands', 'modify',
    '/features', headers, value
  );
}

Send this payload via Eclipse Hono’s HTTP adapter:

echo -e $((0x09EF03F72A)) | curl -i -X POST -u sensor1@DEFAULT_TENANT:hono-secret -H 'Content-Type: application/octet-stream' --data-binary @- http://127.0.0.1:8080/telemetry

The digital twin is updated by applying the script and extracting the relevant values from the bytePayload.

Custom Java mapper

For advanced use cases, implement a custom Java-based mapper by extending AbstractMessageMapper.

Extend AbstractMessageMapper to provide a custom mapper:

public final class FooMapper extends AbstractMessageMapper {

    private static final String MAPPER_ALIAS = "Foo";

    public FooMapper(ActorSystem actorSystem, Config config) {
        super(actorSystem, config);
    }

    private FooMapper(AbstractMessageMapper copyFromMapper) {
        super(copyFromMapper);
    }

    @Override
    public String getAlias() {
        return MAPPER_ALIAS;
    }

    @Override
    public boolean isConfigurationMandatory() {
        return false;
    }

    @Override
    public MessageMapper createNewMapperInstance() {
        return new FooMapper(this);
    }

    @Override
    public List<Adaptable> map(ExternalMessage externalMessage) {
        // TODO implement mapping inbound messages consumed via "sources" to DittoProtocol adaptables
        return null;
    }

    @Override
    public DittoHeaders getAdditionalInboundHeaders(ExternalMessage externalMessage) {
        return DittoHeaders.empty();
    }

    @Override
    public List<ExternalMessage> map(Adaptable adaptable) {
        // TODO implement mapping DittoProtocol adaptables to outbound messages published via "targets"
        return null;
    }

    @Override
    private void doConfigure(Connection connection, MappingConfig mappingConfig,
            MessageMapperConfiguration configuration) {
        // extract configuration if needed
    }
}

Key methods to implement:

• List<Adaptable> map(ExternalMessage message) – inbound mapping (return empty list to drop)
• List<ExternalMessage> map(Adaptable adaptable) – outbound mapping (return empty list to drop)
• String getAlias() – unique mapper alias (must start with uppercase)

To deploy:

1. Add the mapper JAR to the connectivity service classpath (extending Ditto)
2. Register the alias in connectivity configuration
3. Reference the alias in your connection’s mappingDefinitions

For a complete example, see the custom-ditto-java-payload-mapper project.

Further reading

• Connections overview – connection model and configuration
• Header mapping – map external headers
• Ditto Protocol – message format specification