Shore Side Data System (600031 and 900823) : Ingest Architecture This page last changed on Mar 18, 2011 by kgomes.
| This doc has been moved Most of the documentation for SSDS should be on the Google code site now and this page is here: http://code.google.com/p/shore-side-data-system/wiki/TransmogrifyAndIngest
Name | Type | Description
--- | --- | ---
StreamID | java.lang.short | This basically states that the bytes are coming from a SIAM ExportablePacket class. SIAM uses constants defined in the org.mbari.siam.distributed.Exportable.java class to enumerate things like this and the short value for this is always 0x0100. SSDS Doesn't really care so we essentially ignore it.
DevicePacketVersion | java.lang.long | This is the "serialVersionUID" on the class that was used to export the bytes. It is the version of SIAM class that generated the byte array. SSDS Does not really care and as of this writing, it is always 0.
SourceID | java.lang.long | The ID of the device that the message was generated by.
Timestamp | java.lang.long | Epoch milliseconds (number of elapsed milliseconds since 1/1/1970 00:00:00) that the packet was generated by the device
SequenceNumber | java.lang.long | A number that is supposed to show the order of generation of packets from the device
MetadataRef | java.lang.long | This is the sequence number of the packet that contains the metadata that describes the information in this packet
ParentID | java.lang.long | The ID of the device that the generated device was attached to when it generated the packet
RecordType | java.lang.long | The type of record that this packet contains: 0 = MetadataPacket 1 = Non-MetadataPacket (Data and other)
SecondStreamID | java.lang.short | This defines the type of DevicePacket that was used to construct the byte array. The values are as follows: MetadataPacket = 0x101 SensorDataPacket = 0x102 DeviceMessagePacket = 0x103 SummaryPacket = 0x102 (same as SensorDataPacket)
SecondPacketVersion | java.lang.long | This is the "serialVersionUID" on the class that was used to export the bytes. It is the version of SIAM class that generated the byte array. As of this writing, it is the same as the DevicePacketVersion. Since currently it is always 0, SSDS ignores it.
FirstBufferLength | java.lang.int | This is the length of the array that holds the bytes of the first buffer
FirstBuffer | java.lang.byte [] | This is the bytes array that represents the first buffer
SecondBufferLength | java.lang.int | This is the length of the array that holds the bytes of the second buffer.
SecondBuffer | java.lang.byte [] | This is the array that holds the bytes of the second buffer.
Now, in order to handle both types of inputs in Transmogrify (DevicePackets and BytesMessage structure), Transmogrify would take both and convert to a common format that would contain the information to cover both types of messages. Since the BytesMessage structure encompasses all the information in the DevicePacket, we simply used that byte structure and in Transmogrify, a DevicePacket is converted to a SSDSDevicePacket which is then converted to the same BytesMessage structure using the SSDSDevicePacket.convertToPublishableByteArray method. So at the end of the Transmogrify process, we have on byte array that is in the form of the diagram above that will then be used to publish a message to the next component which is Ingest. Transmogrify takes the SIAM byte array structure and converts it to the SSDS native byte array structure: Notes on the conversion: The DevicePacketVersion, SecondStreamID, and SecondPacketVersion are used to determine the correct packetType (although, right now, DevicePacketVersion and SecondPacketVersion are ignored). For MetadataPackets, the packetType is 1. For SensorDataPackets, the packetType is 0 (SummaryPackets come across as SensorDataPacket and are differentiated by their recordType). For DeviceMessagePackets, the packetType is 4. If the incoming packet is a MetadataPacket, the packetSubType is set to 0. Otherwise, it is set to the RecordType field. The RecordType is set to zero if the packet is a MetadataPacket and set equal to the RecordType from SIAM if not a MetadataPacket. The MetadataSequenceNumber is calculated depending on the device, it's parent, and the XML that is in it's payload. There is a component called the SIAMMetadataTracker that keeps track of this information and looks for real XML changes which is what should fire a change in metadata. The buffers are swapped if it is a MetadataPacket. It always seemed to logical to do it that way. This timestamp (epoch milliseconds) is split into seconds and nanoseconds.
| Message Size Limitation! Please note that because byte arrays are limited to 32 bit sizes, the largest payload of a message that can be converted by SSDS is 2GB. While this does not seem like a major restriction, it can be hit if somebody is using straight JMS messaging (or other) and makes a payload bigger than 2GB. SSDS will just ignore such a message.
##### Ingest Packet Structure So now we have all messages coming into Ingest in a format that SSDS is expecting (i.e. that matches the SSDS view of the world). For the diagram in the previous section, the attributes in the SSDS Bytes Array are:
Attribute | Type | Description
--- | --- | ---
sourceID | java.lang.long | This is what is known as the SSDS ID for the device (i.e. DeviceID) that actually generated the packet of information.
parentID | java.lang.long | This is the SSDS ID for the parent that the device was connected to when it sent the packet. If the ID is zero (0), then the generating device was not connected to a parent.
packetType | java.lang.int | This is the "Type" of packet that is being sent. It is basically an enumerated list the with following context: 0 = Data Packet 1 = Metadata Packet 2 = 3 = 4 = Device Message Packet
packetSubType | java.lang.long | This is the equivalent of the "recordType" listed in the Transmogrify component. It is used to provide the hook to tell the client applications what type of record this is that was sent. It really only has meaning in the context of data packets as a device can often send data packets of different formats. This basically tells the application which record form is being sent in this packet.
metadataSequenceNumber | java.lang.long | Also referred to as dataDescriptionID
dataDescriptionVersion | java.lang.long |
timestampSeconds | java.lang.long) |
timestampNanoseconds | java.lang.long |
sequenceNumber | java.lang.long) |
bufferLen | java.lang.int |
bufferBytes | java.lang.byte[bufferLen] |
bufferTwoLen | java.lang.int |
bufferTwoBytes | java.lang.byte[bufferTwoLen] |
The Ingest Message Driven Bean (MDB) then takes that byte array and using a PacketOutput class that corresponds to the correct source ID, metadataSequenceNumber, packetSubType, and parentID, it writes the packet to disk. It then uses a PacketSQLOutput to write that same packet to a table in the database. ##### Packet Translations So, through all this, there are basically four representations of data packets in the SSDS ecosystem: SIAM Device Packet (and its sub classes MetadataPacket, SensorDataPacket, DeviceMessagePacket) SSDSDevicePacket (and its sub class SSDSGeoLocatedDevicePacket) SIAM Byte array (from Exportable class) SSDS Byte array Here is a diagram of these various forms of data And the translation rules (some of these may seem very strange for legacy reasons). ###### DevicePacket to SSDSDevicePacket This translation is done in the constructor of SSDSDevicePacket which takes in a DevicePacket
DevicePacket | Translation Rule | SSDSDevicePacket
--- | --- | ---
sourceID | direct copy | sourceID
systemTime | direct copy | systemTime
sequenceNo | direct copy | sequenceNo
metadataRef | direct copy: metadataRef->metadataRef metadataRef->metadataSequenceNumber metadataRef->dataDescriptionID | metadataRef metadataSequenceNumber dataDescriptionID
parentId | direct copy: parentId->parentId parentId->platformID | parentId platformID
recordType | MetadataPacket: recordType to 0 Other: direct copy | recordType
| If MetadataPacket, packetType = 0 If SensorDataPacket, packetType = 1 If DeviceMessagePacket, packetType = 2 | packetType
firstBufferLength | ignored |
firstBuffer | First buffer depends on which type of packet If MetadataPacket, copy "bytes" buffer If SensorDataPacket, copy "dataBuffer" If DeviceMessagePacket, copy "message" | firstBuffer
secondBufferLength | ignored |
secondBuffer | Only exists if MetadataPacket and will copy over "cause" buffer | secondBuffer
###### DevicePacket to SIAM Byte Array This is done by the SIAM Exportable Packet class
DevicePacket | Translation Rule | SIAM Byte Array
--- | --- | ---
| This is a static value that is set to indicate the byte array is a DevicePacket and is set to 0x0100 | EX_DEVICEPACKET
| This is just the serial version UID of the class which is always 0 | serialVersionUID
sourceID | direct copy | sourceID
systemTime | direct copy | systemTime
sequenceNo | direct copy | sequenceNo
metadataRef | direct copy | metadataRef
parentId | direct copy | parentId
recordType | direct copy | recordType
| This is set based on what type of packet: If MetadataPacket, set to 0x0101 If SensorDataPacket, set to 0x0102 If DeviceMessagePacket, set to 0x0103 | EX_XXXXXXPACKET
| This is just the serial version UID of the class which is always 0 | serialVersionUID
firstBufferLength | direct copy (see note for first buffer) | firstBufferLength
firstBuffer | This depends on the type of packet: If MetadataPacket, "cause" bytes are copied over If SensorDataPacket, "dataMessage" bytes are copied over If DeviceMessagePacket, "message" bytes are copied over | firstBuffer
secondBufferLength | only valid with MetadataPacket, but is copied directly over | secondBufferLength
secondBuffer | only valid with MetadataPacket, and the "buffer" bytes are copied over | secondBuffer
###### SSDSDevicePacket to SIAM Byte Array Originally done in TransmogrifyMDB by calling SSDSDevicePacket.convertToPublishableVersion3ByteArray before passing byte array to method to translate to SSDS format and then send to ingest
SSDSDevicePacket | Translation Rule | SIAM Byte Array
--- | --- | ---
| This is a static value that is set to indicate the byte array is a DevicePacket and is set to 0x0100 | EX_DEVICEPACKET
| This is just the serial version UID of the class which is always 0 | serialVersionUID
sourceID | direct copy | sourceID
systemTime | direct copy | systemTime
sequenceNo | direct copy | sequenceNo
metadataSequenceNumber | direct copy | metadataRef
parentID | direct copy | parentID
recordType | If packetType = 0, set recordType = 0 If packetType = 1, set recordType = recordType If packetType = 2, set recordType = recordType | recordType
| This is set based on what type of packet: If packetType = 0, set to 0x0101 If packetType = 1, set to 0x0102 If packetType = 2, set to 0x0103 | EX_XXXXXXPACKET
| This is just the serial version UID of the class which is always 0 | serialVersionUID
| This depends on the type of packet: If packetType = 0, set to length of "otherBuffer" If packetType = 1, set to length of "dataBuffer" If packetType = 2, set to length of "dataBuffer" | firstBufferLength
other/dataBuffer | This depends on the type of packet: If packetType = 0, set to bytes from "otherBuffer" If packetType = 1, set to bytes from "dataBuffer" If packetType = 2, set to bytes from "dataBuffer" | firstBuffer
| This only exists if it is packetType = 0 then it is set to the length of the "dataBuffer" | secondBufferLength
dataBuffer | This only exists if it is packetType = 0 then it is set to the byte from the "dataBuffer" | secondBuffer
##### SSDSDevicePacket to SSDS Byte Array Originally done in SSDSDevicePacket.convertToVersion3ByteArray
SSDSDevicePacket | Translation Rule | SSDS Byte Array
--- | --- | ---
sourceID | direct copy | sourceID
systemTime | ignored during the translation directly, but used through getter methods for seconds and nanoseconds |
timestampSeconds | direct copy (note that this is sort of a direct copy, there are getter methods on SSDSDevicePacket that convert the systemTime to seconds and nanoseconds when called). | timestampSeconds
timestampNanoseconds | direct copy (note that this is sort of a direct copy, there are getter methods on SSDSDevicePacket that convert the systemTime to seconds and nanoseconds when called). | timestampNanoseconds
sequenceNo | direct copy | sequenceNumber
metadataRef | ignored |
parentID | ignored |
recordType | If packetType = 0, set packetSubType to 0, otherwise set to recordType | packetSubType
packetType | Depends on packetType: If packetType = 0, set to 1 If packetType = 1, set to 0 If packetType = 2, set to 4 | packetType
metadataSequenceNumber | direct copy | metadataSequenceNumber
dataDescriptionVersion | direct copy | dataDescriptionVersion
platformID | direct copy | parentID
| copy length of dataBuffer | firstBufferLength
dataBuffer | direct copy | firstBuffer
| copy length of otherBuffer | secondBufferLength
otherBuffer | direct copy | secondBuffer
###### SIAM Byte Array to SSDS Byte Array Originally done in TransmogrifyMDB in checkAndPublishBytes method
SIAM Byte Array | Translation Rules | SSDS Byte Array
--- | --- | ---
EX_DEVICEPACKET | ignored |
serialVersionUID | ignored |
sourceID | direct copy | sourceID
| Depending on EX_XXXXXXXPACKET: If MetadataPacket, set packetType to 1 If SensorDataPacket, set packetType to 0 If DeviceMessagePacket, set packetType to 4 | packetType
| This was set using the SIAMMetadataTracker that tried to keep track of real version numbers based on XML in payload | metadataSequencNumber
systemTime | Split into timestampSeconds and timestampNanoseconds | timestampSeconds timestampNanoSeconds
sequenceNo | direct copy | sequenceNumber
metadataRef | direct copy | dataDescriptionVersion
parentId | direct copy | parentID
recordType | If MetadataPacket (determined from EX_XXXXXXXPACKET), recordType set to 0, otherwise set to recordType | packetSubType
EX_XXXXXXXPACKET | ignored in storage, but used in logic |
serialVersionUID | ignored |
first/secondBufferLength | If MetadataPacket (determined from EX_XXXXXXXPACKET), firstBufferLength is set to secondBufferLength so we can flip the "cause" and "buffer" bytes because it just made more sense since the cause was rarely populated. Otherwise set to firstBufferLength | firstBufferLength
first/secondBuffer | If MetadataPacket (determined from EX_XXXXXXXPACKET), firstBuffer is set to secondBuffer so we can flip the "cause" and "buffer" bytes because it just made more sense since the cause was rarely populated. Otherwise set to firstBuffer | firstBuffer
firstBufferLength | If MetadataPacket (determined from EX_XXXXXXXPACKET), secondBufferLength is set to firstBufferLength to flip "cause" and "buffer" bytes | secondBufferLength
firstBuffer | If MetadataPacket (determined from EX_XXXXXXXPACKET), secondBuffer is set to firstBuffer to flip "cause" and "buffer" bytes | secondBuffer
##### Exploration of Upgrade of Ingest/Transmogrify to AMQP In an effort to allow non-Java clients to send data to SSDS in the form of messages and to upgrade the messaging system to a technology that is more scalable and higher performance, an investigation of AMQP implementations was done. Qpid Exploration Attachments: SSDS JMS (text/xml) SSDS JMS (text/xml) SSDS JMS (text/xml) SSDS JMS (text/xml) SSDS JMS (text/xml) SSDS JMS (text/xml) SSDS JMS (text/xml) SSDS JMS (text/xml) SSDS JMS (text/xml) SSDS JMS (text/xml) SSDS JMS (text/xml) SSDS JMS (text/xml) SSDS JMS (text/xml) SSDS JMS (text/xml) SSDS JMS (text/xml) SSDS JMS (text/xml) SSDS JMS (text/xml) SSDS JMS (text/xml) SSDS JMS (text/xml) SIAM Externalizable Structure (text/xml) SIAM Externalizable Structure (text/xml) SIAM Externalizable Structure (text/xml) SIAM Externalizable Structure (text/xml) SIAM Externalizable Structure (text/xml) SIAM Externalizable Structure (text/xml) SIAM Externalizable Structure (text/xml) Device Packets.jpg (image/jpeg) Device Packets.jpg (image/jpeg) Device Packets.jpg (image/jpeg) SSDSByteArrayFormat (text/xml) SSDSByteArrayFormat (text/xml) SSDSByteArrayFormat (text/xml) SSDSByteArrayFormat (text/xml) SSDSByteArrayFormat (text/xml) SSDSByteArrayFormat (text/xml) SSDSByteArrayFormat (text/xml) SSDSByteArrayFormat (text/xml) SSDSByteArrayFormat (text/xml) SSDSByteArrayFormat (text/xml) SSDSByteArrayFormat (text/xml) SSDSByteArrayFormat (text/xml) Transmogrify Steps.jpg (image/jpeg)

Document generated by Confluence on Feb 04, 2026 08:56