Working with the Radar HDF5 Files

These HDF5 files were generated from wireshark network captures of the radar data, and this file will show an example analysis of a single file, exploring the data contained within.
close all; clear;
Table of Contents
Goals HDF5 Radar Files Loading parts of the HDF5 File AHRS Data Radar Reports Scan Display Next Steps

Goals

I have developed a program in C that converts the network capture files into archival, self describing data files using the HDF5 format. These files are heirarchal, and is a format designed for large dataset analyses like the ones we want to acheive with the radar data. HDF5 files can be ingested by any software toolset that has supported libraries, most notably MATLAB, Python, C and Fortran. If you are arriving at this document without being at the repository, you can find the conversion tools, MATLAB and python scripts (including this one) here:
https://bitbucket.org/mbari/wec-pcap-utils/src/master/

HDF5 Radar Files

Assuming you are in the repo, let's select a file relative to this script. You may have to change this for non-posix.
hdf_file = '../data/mbariAtSeaWecRadar_00315_20210928171518.h5';
% Let's use matlab's tools to print out the information in this file
h5disp(hdf_file,'/','min')
HDF5 mbariAtSeaWecRadar_00315_20210928171518.h5 Group '/' Group '/ahrs' Dataset 'altitude' Dataset 'dVelocity' Dataset 'eVelocity' Dataset 'latitude' Dataset 'longitude' Dataset 'nVelocity' Dataset 'packet_time' Dataset 'pitchAngle' Dataset 'rollAngle' Dataset 'timestamp' Dataset 'xAccel' Dataset 'xRateCorrected' Dataset 'xRateTemp' Dataset 'yAccel' Dataset 'yRateCorrected' Dataset 'yawAngleTrue' Dataset 'zAccel' Dataset 'zRateCorrected' Group '/radar_a' Dataset 'angle' Dataset 'range' Dataset 'scan' Dataset 'time' Group '/radar_b' Group '/reports_a' Dataset 'command' Dataset 'gain' Dataset 'interference_rejection' Dataset 'rain' Dataset 'range' Dataset 'sea' Dataset 'sea_auto' Dataset 'target_boost' Dataset 'target_expansion' Dataset 'time' Group '/reports_b' Dataset 'command' Dataset 'gain' Dataset 'interference_rejection' Dataset 'rain' Dataset 'range' Dataset 'sea' Dataset 'sea_auto' Dataset 'target_boost' Dataset 'target_expansion' Dataset 'time'

Loading parts of the HDF5 File

You can learn more about the HDF file format at https://portal.hdfgroup.org/display/HDF5/HDF5
One thing that is great is that you only need to load the items into memory that you want to work with, so we'll step into loading something familiar, the AHRS data.

AHRS Data

Load the ahrs data into memory from the hdf5 file.
ahrs.t = h5read(hdf_file, '/ahrs/timestamp');
ahrs.r = h5read(hdf_file, '/ahrs/rollAngle');
ahrs.p = h5read(hdf_file, '/ahrs/pitchAngle');
ahrs.y = h5read(hdf_file, '/ahrs/yawAngleTrue');
% Let's convert the timestamp into something more plottable.
ahrs.dt = datetime(ahrs.t, 'ConvertFrom','epochtime');
Plot this 5 minute window of time.
figure();
subplot(311);
plot(ahrs.dt,ahrs.r);
title(hdf_file,'Interpreter','none');
ylabel('roll');
subplot(312);
plot(ahrs.dt,ahrs.p);
ylabel('pitch');
subplot(313);
plot(ahrs.dt,ahrs.y);
ylabel('yaw');
There is obviously more data to explore in the AHRS records, but mostly we need the yawAngleTrue to offset the plots of the incoming radar data. We get our first indication in this file that heading is ±180˚ instead of the typical 0-360˚ range.
Also, I have started loading functions for the three main types, let's test the ahrs load function:
ahrs = load_radar_ahrs(hdf_file);
head(ahrs)
ans = 8×19 table
 altitudedVelocityeVelocitylatitudelongitudenVelocitypacket_timepitchAnglerollAngletimestampxAccelxRateCorrectedxRateTempyAccelyRateCorrectedyawAngleTruezAccelzRateCorrecteddt
1-33.7500-0.5078-0.132836.7431-121.88110.17191.6329e+09-1.3568-0.74711.6329e+090.07486.517624.15770.0598-0.1346140.3723-0.99401.038229-Sep-2021 00:15:18
2-33.7500-0.5156-0.093836.7431-121.88110.06251.6329e+09-1.4447-0.12631.6329e+090.09285.787024.15770.0314-1.5381140.4602-1.01901.019029-Sep-2021 00:15:18
3-33.7500-0.53910.007836.7431-121.88110.03121.6329e+09-1.71390.30211.6329e+090.09773.441524.15470.0034-3.5953140.5371-1.03490.826729-Sep-2021 00:15:19
4-33.7500-0.57810.093836.7431-121.8811-0.05471.6329e+09-2.20830.53281.6329e+090.08421.134324.1516-0.0256-5.9024140.6030-1.04370.596029-Sep-2021 00:15:19
5-33.5000-0.57810.148436.7431-121.8811-0.00781.6329e+09-2.86190.49991.6329e+090.0589-1.345824.1516-0.0470-7.2867140.6689-1.06050.730629-Sep-2021 00:15:19
6-33.5000-0.64060.242236.7431-121.8811-0.05471.6329e+09-3.60350.25821.6329e+090.0214-3.191524.1486-0.0577-7.4020140.7349-1.06780.903629-Sep-2021 00:15:19
7-33.5000-0.71090.335936.7431-121.8811-0.07811.6329e+09-4.2792-0.12081.6329e+09-0.0186-4.114424.1486-0.0641-6.2100140.8173-1.07971.019029-Sep-2021 00:15:19
8-33.5000-0.72660.375036.7431-121.8811-0.03911.6329e+09-4.7461-0.57131.6329e+09-0.0592-4.479724.1486-0.0607-3.3261140.9326-1.08281.307429-Sep-2021 00:15:19

Radar Reports

The reports channel tells us at 1hz the operational state of the radar. For the most part these are changing very little. This is our first example of of a loading function to simplify the import process.
reports_a = load_radar_reports(hdf_file,'a');
% Let's summarize the data
head(reports_a)
ans = 8×10 table
 timecommandrangegainrainseainterference_rejectionsea_autotarget_boosttarget_expansion
11.6329e+0919650001552660011
21.6329e+0919650001552660011
31.6329e+0919650001552660011
41.6329e+0919650001552660011
51.6329e+0919650001552660011
61.6329e+0919650001552660011
71.6329e+0919650001552660011
81.6329e+0919650001552660011
headers = reports_a.Properties.VariableNames;
for i = headers
rpt_mean = mean(double(reports_a{:,i}));
rpt_std = std(double(reports_a{:,i}));
disp(sprintf("Mean: %10.1f\tStd: %5.2f\t %s", rpt_mean, rpt_std, i{1}));
end
Mean: 1632874668.6 Std: 86.75 time Mean: 196.0 Std: 0.00 command Mean: 5000.0 Std: 0.00 range Mean: 155.0 Std: 0.00 gain Mean: 2.0 Std: 0.00 rain Mean: 66.0 Std: 0.00 sea Mean: 0.0 Std: 0.00 interference_rejection Mean: 0.0 Std: 0.00 sea_auto Mean: 1.0 Std: 0.00 target_boost Mean: 1.0 Std: 0.00 target_expansion
I'm still working on decoding these, for now we will move forward.

Scan Display

Load in scan data and plot one full rotation. Its worth describing the nomenclature in the variables. The radar collects data in a polar coordinate system, for each "scan", there is a primary angle and a total range that return intensity is sampled 1024 times along. Hence, each row of the datatype as compiled in the table below contains one scan. Each full revolution of the radar is 2048 scans, or spokes.
scans_a = load_radar_scans(hdf_file,'a');
head(scans_a)
ans = 8×4 table
 timeanglerangescan
11.6329e+09295.5762808151515150000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000102151515151515151515151515151515151515070000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
21.6329e+09295.752080815151515000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000080151515151515151515151515151515151515050000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
31.6329e+09295.927780815151515000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000060151515151515151515151515151515151115030000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
41.6329e+09296.1035808151515150000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000101581515151515151515151515151515712000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
51.6329e+09296.27938081515151500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000083151215151515151515151515101248000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
61.6329e+09296.45518081515151500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000040108121212121212121210116815000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
71.6329e+09296.630980815151515000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006398999988673401000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
81.6329e+09296.806680815151515000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002054555544230000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
% plot 1 revolution, each scan is 2048 lines
idx = 1:2048;
single_revolution = scans_a(idx,:);
rho = 1:1024; %create a range vector for plotting, units ignored for now.
theta = ones(1,1024); % a ones vector we will change each loop through scan data.
%now build a plot one scan at a time, let's plot 1 scan
figure();
for i = idx
ang = deg2rad(single_revolution.angle(i))*theta;
col = single_revolution.scan(i,:);
polarscatter(ang,rho, 1,col,'filled');
hold on;
end
Now processing the whole file doesn't make a ton of sense, that plot above took 30 seconds to make, so it isn't computationally efficient whatsoever. This file has about 180 full revolutions, but it does illustrate how to get a first pass set of images together.

Next Steps