Skip to content

Dataprobe Users and Developers Guide

Author: Rich Schramm Revision: 2.1.1 Date: 18-Jun-2020

Revision history

  • 1.0 22-Apr-20 original.
  • 1.1 18-Jul-20
  • 2.0 22-Apr-2020 Initial rewrite after conversion to Xojo
  • 2.1 17-Jun-2020 Rework the Preferences and Installations sections
  • 2.1.1 18-Jun-2020 Minor correction to path to executable in ‘StartingDataprobe section.

Introduction

This guide is intended to provide user and developer guidance on the Dataprobe application to support future maintenance.

Successful core data monitoring entails having fundamental knowledge of how the data acquisition sub-systems interact with each other, what some of the historical instrument and sub-system failure modes are, and what clues the system can give us to infer that it is working properly. Many of the checks that the system developers and expert support staff routinely use to monitor and debug the system have been coded into an application named Dataprobe.

This program automatically performs over seventy specific tests on the core data-streams every two seconds and presents the results through a very simple and highly visual computer display. Dataprobe also provides on-screen troubleshooting help associated with each test. The program can be run from any PC on the vessel or remotely over the network when the ship is in range.

Dataprobe puts into code many of the common checks we use to troubleshoot datastream issues. Thus we use it it to infer the overall the health of a system that dataprobe itself requires to be running at some minimal level.

Background

The original dataprobe was written by Rich Schramm in Visual Basic 4 sometime around the delivery of the R/V Western Flyer (1996?). It has been in continuous use on three MBARI ships and used to support three ROV’s (Western Flyer, Pt. Lobos, Rachel Carson, Ventana, Tiburon and Doc Ricketts). While its basic form and function has not changed it has been tweaked and improved over the decades as hardware, ships and ROV’s changed. Eventually VB4 aged to VB5, then VB6. VB6 was dropped by Microsoft in 2008 however we kept it ‘alive’ through use of Virtual Machines.

In 2020, Dataprobe was updated and ported to XOJO. XOJO was picked as it was ‘similar’ to VB, was being used in numerous other MBARI GUI and console applications (The Oasis5 apps, Ventana and DocRickets Divelogs, ModBusTester etc) and is cross-platform (OSX, Windows, Linux).

The Data logging is performed by programs running on the Linux servers currently named rcnavproc1 or wfnavproc1. For the purpose of this document – they will be simply be called “navproc”. All data inputs to the loggers are through the MBARI navproc/LCM facilities.

There are data logging programs running on navproc for each of the major sub-systems: navigation, video and rovctd. A unique feature of the logging system is that in addition to writing ascii csv strings of data to files, each logger also writes it’s string to a broadcast UDP network port. A companion program to each logger listens for this broadcast and also for requests from network client programs (such as dataprobe). It is these messages on the network which Dataprobe relies on to monitor the health of the system. This two-step process is used as broadcast UDP datagrams do not pass beyond the router, whereas the non-broadcast UDP Servers (bcserver) can be reached from anywhere, even ship-to-shore if the network ports are enabled by IS.

In addition to the standard core data loggers (nav, ctd, video), Dataprobe also relies on a special logger instance named dataprobelogr that has been set up on to provide additional information that it needs to be able to monitor the core loggers themselves, as well as numerous data items useful for debugging that are not logged within the standard core variables. Dataprobelogr data is provided via the same UDP mechanism described above.

Ship & ROV Data Distribution

The MBARI navproc/LCM is the data distribution mechanism for all for the data logged by the system. It is basically a networked based publish/subscribe system, meaning there are tasks which can register with LCM to be providers of named items, and tasks which register to as subscribers to items (or both). Subscribers can either register to be notified (or ' triggered') when an item updates, or they can simply poll the LCM publisher periodically to see what the 'current' value of an item is. 'Current' is used loosely as what is actually provided is whatever the value was the last time it was written to LCM system which could be days or weeks old. A robust consumer therefore needs to examine the status and time associated with the item to fully determine the item’s age and validity.

Starting Dataprobe

Dataprobe is currently (5/2020) installed on ‘copilot’ and the science PC 'lava' in the ROV Control Room of Western Flyer and on ‘Infernalis’ for Rachel Carson.

Due to how easy it is for users to change the pc ‘environments’, steps 1 and/or 2 might not apply.

Dataprobe can be started in one of these usual PC ways:

  1. There should be a shortcut installed on the Desktop (look for an icon named Dataprobe) and double click it.
  2. Start – Programs –Dataprobe – Dataprobe.exe
  3. The program is installed in C:/Users/xxxx/Dataprobe/dataprobe.exe – cd there and double-click it.

You should see the main panel that looks like this:

dp-ug1

Each button on this panel is related to a major data-logging sub-system. Clicking on the button with the mouse will pop-up a detailed panel for that sub-system. The color of each button on this panel indicates a summary of success (green) or failure (red or yellow) of all controls on the corresponding detail panel. Example detail panel is shown below. By right-clicking in the grey margins you can enable/disable the display of test result icons in the windows system tray.

dp-ug2

The checkbox next to a control can be used to 'acknowledge' any test failure. If, for example, there is no differential GPS signal available, the differential mode will fail and the button will go red at the detail control as well as on the main panel. If you are working in a region where DGPS is not available (i.e.. its not really an error condition) - it is useful to 'acknowledge' this by un-checking the test. The button will then appear yellow (caution). And, assuming all other test buttons on the sub-panel are green, then the main panel will also show yellow. If any other test on this panel goes red - the summary button will then turn from yellow to red indicating some other problem has occurred.

Dataprobelogr

Dataprobe relies on a special logger named dataprobelogr that has been set up on navproc to provide the information Dataprobe needs to be able to monitor the core loggers. Dataprobelogr is started automatically shortly after the server is booted by the Unix cron facility. Its existence is checked for by scripts which are run once every few minutes. If a script executed by cron cannot detect its process, it automatically attempts to restart it.

Note: The Navproc code must be running on the corenav server (via Procman Sheriff)

Starting dataprobelogr

It is always best to let cron handle the actual startup of the loggers. However it can be started and stopped manually. To do so:

  1. Log in as ops
  2. cd to the /home/ops/corelogging/[rc|sim|wf]/scripts directory
  3. Run ./startLogr dataprobelogr
  4. You can then cd to the /home/ops/corelogging/[rc|sim|wf]/data directory and look for the YYYYDDDdataprobelogr.dat file that is updating regularly. You can then run tail -f YYYYDDDdataprobelogr.dat and watch as new data is written to the file.
  5. You can now safely log out (the cron job and applications will continue to run).

Stopping dataprobelogr

The presence of the dataprobelogr application is tested for once every few minutes via a cron job owned by user ops. If missing, it is restarted automatically. It typically runs all of the time, however for maintenance purposes it may be necessary to stop it manually for a period of time.

Note the stopLogr command can be used to shutdown the special loggers such as dataprobelogr. However you should not need to do this and are strongly discouraged from interfering with them! Also, if the crontab is still enable, even if you shutdown a logr, it will be started automatically by cron. You can disable this in cron if you do not want that to happen.

For The Developer

Preliminaries

Your Xojo Development System

Follow standard Xojo installation for your system, then add the required 3rd party 64-bit versions plugins to your Xojo “plugins” directory. See section “THE MBS NOTIFICATION PLUGIN” in this document.

Installation of the Development Project

The source code tree is stored in Bitbucket for version control. It is a private repo that requires a BitBucket account that is part of the Administrators or Information Engineering groups. It is in the Dataprobe repository within the MBARI/Core Data project folder.

Familiarity with basic git commands is assumed. Basically you want to clone the repo to your users home directory (e.g. for OSX for user xxxx would be /Users/xxxx).

Using a git clone command on the repository results in the directory structure:

/Users/xxxx/dataprobe
    Readme.md
    /src
    /doc
    /Resources

The src folder will contain various xojo code modules and subfolders. The important one to point out for setting up your development system is the Dataprobe.xojo_project. This is the top-level file that you will open in the Xojo IDE. Also, there is a “Builds – Dataprobe(vX.X.X)” folder. This is where Xojo builds the executable for the selected target operating systems.

Before starting, you will need to have a valid license for the MBS plugin. If you purchase the MBS plugin or get the license transferred to you from someone else, you will end up with a license that looks something like:

dim s as string = decodeBase64("****************************", encodings.UTF8)
dim p as string = decodeBase64("****************", encodings.UTF8)
dim n as string = decodeBase64("********************", encodings.UTF8)
dim e as integer = ******
dim t as string = decodeBase64("************************************************************", encodings.UTF8)

if not registerMBSPlugin(n, p, e, s+t) then  
  MsgBox "MBS Plugin serial not valid?"  
end if

Warning

You must take steps to protect the license key for the Monkeybread Software 3rd party Xojo plugin. Do not check in the license code you entered back into the Git remote origin server.

Once you have this, open the Dataprobe project and navigate to App > Methods > CheckMBSLicense and replace the commented out code with your license code. You should then be able to start the application by clicking on the Run green arrow. The first times this runs on your machine, it will create a Dataprobe/Logs directory and a Dataprobe/Resources folder in your home directory. It will also install templates and help files in the Dataprobe/Resources directory to get you started. You then need to open the Dataprobe/Resources/Dataprobe.pref file in a text editor. You will need to change the names of the various servers to point to the location where the navproc, logrs, and bcservers are running as well as what ports it needs to send requests to. This information can be located in the /home/ops/corelogging/[rc|sim|wf]/cfgs/core_logging.ini on the server that is running those services. It will tell you which ports you need to enter into the Dataprobe.pref file. Once that update is done, you should be able to start/re-start Dataprobe from the XOJO IDE and it should start to send UDP requests to the bcservers and processing the data returns.

System Design

XOJO GUI applications use a components and an events (callbacks) model (similar to VB4 and VB6):

  • GUIs are composed of objects (windows, buttons, text fields, checkboxes, sockets etc)
  • GUI component objects can have properties (ints, strings, floats, arrays and other objects) and methods.
  • GUI objects have defined event callbacks (actions for button press, textchange, socket data-available etc.) you can choose to respond to or not. You implement the functions you want to respond to in the XOJO IDE.
  • You can also define (write) your own classes that implement objects, properties, methods, and events. You can ‘compose’ other GUI objects by combining GUI objects and encapsulate specific behavior within them.
  • Your application is designed inside of the XOJO IDE by coding up all of these pieces and compiling it into the finished executable. Objects encapsulate specific behavior and state within them.
  • The ‘top level’ object in the Xojo project is the App (application object) that has an Open() event method. This is where all initialization happens prior to opening the default window.

Building window apps with a large number of components and wiring their events up in code at design time can get tedious and can become a maintenance issue. The Xojo component Event Model makes it difficult to separate the state (data/model) from the view (gui components) and the events (controller). Components, events and functions ‘know too much’ about each other to keep code clean (they become tightly coupled).

To address this, Dataprobe adopts a more Model-View-Controller (MVC) approach using both the Xojo event model and a Notification/Observer messaging service to communicate state changes between the data model and the view components. This service is implemented using a 3rd-party Xojo plugin class purchased from Monkeybread Software (MBS) (see section titled "The MBS Notification Plugin“)

Generally speaking, when new data comes in that changes the ‘Data Model’, the Model will publish a uniquely named and typed message (plus a value) to registered subscribers. Components on windows, register and watch for the specific messages they care about. Standard Xojo event callback functions (like button press actions etc,) may also include code to publish their own named messages back out to registered subscribers (i.e. back to the Controller or the Model)

It is important to realize there are both Xojo Events and Notification messages flying about in the running application. For example, a window checkbox click Xojo event may perform some action in the GUI world (like change a button color) and then might also publish a message as a ‘notification’ to inform the controller and data model that the View has changed.

Unfortunately the real world is messy – especially when writing UI’s. My use here of MVC is as a logical construct. It helps organize the application structure to improve maintainability. There are sure to be plenty of ‘violations’ of an OO purists view of MVC in Dataprobe. Separating the data model from the view and from the controller is a good design objective. However implementing that in a GUI tool that relies on a more tightly coupled Component+Properties - View+CallbackEvents still require us to manage the messy bits someplace in the application. The goal of this document is show you where this happens so you can maintain and extend Dataprobe into the future.

Take some time to examine the visible running Dataprobe GUI app. You will notice the following:

  • The Main window consists of (currently five) top-level command buttons. Each button represents a set of tests that are related to the health of some logical subset of the core data logging system. Button color reflects the rolled-up pass/fail state of all the tests of that “TestSet”.
  • Clicking a top-level command button raises a floating window panel that graphically displays the state all of the individual Tests within that particular TestSet
  • Each individual Test is displayed as a compound component consisting of a checkbox, labeled command button and text-field. It displays green for passed, red or yellow for failed.
  • The checkbox allows the user to acknowledge a failing test, changing its button color to yellow. If any individual test fails (red or yellow), that information cascades up to the top-level command button causing its color to reflect the ‘worst-case’ of all the tests on its sub-panel (red is worse than yellow, green is all good).
  • Clicking a Test command button pops-up a Help window that displays a text file describing the item, its importance and and some suggestions on how to debug the problem.

The data model (or state) reflects this graphical view of the system in code as data objects. The top-level of the data model is implemented in a class called “Model”. This class has two important properties, TestArray and TestSetArray which are object collections of type “Test” and “TestSet” respectivley. (A third property of Model is a dictionary to support lookup of a Help files using named tests but that is minor to this discussion.)

The Data Model is built dynamically from a JSON text file which is read on startup. This file typically resides in the Dataprobe/Resources sub-folder in the user home directory. It has a structure similar to what is shown below in Figure 1. Where the JSON resides and how it gets found and read is covered in the section on Preferences elsewhere in this document but for now, just know that the model is constructed at the end of the App.Open() event method.

{
  “TestSets”: [
    {
      “TestSetName”: “ShipNavTestset”, 
      "Tests": [
        {
          "TestType": “MIN_MAX”,
          "Name": "ShipLoggerAge",
          "HelpHint": "Approx fix age in seconds",
          "HelpFile": "Help/RCARSON/ShipNav/loggerAge.txt",
          "Max": 10,
          "Min": -5
        },
        {
          "TestType": “DATE_TIME”,
          "Name": "ShipLogrDatetime",
          "HelpHint": "Datetime of last logr record",
          "HelpFile": "Help/RCARSON/ShipNav/datetime.txt",
          "Max": 0,
          "Min": 0
        }
      ]
    }
  ]
}

Figure 1 - JSON Data Model

The above represents the data model as an array of “TestSets”. This abbreviated example shows one test set element identified as the ShipNavTestset. Besides its identifying name, each testset contains an array of specific tests. Each test must be one of the predefined TestTypes, along with other parameters appropriate to the type of the test. Each test also contains a relative path to the help file that is used to document the test.

Two things that are critical to understanding and that require developer-discipline to maintain are the “TestSetName” and the test “Name” attributes. These two identifiers must be kept unique across the full JSON initialization file. They are the messy glue that tie the data model, and message notifications, etc to the individual named hard-coded Xojo window components. More details to come, but for now there are named Xojo controls that are hardwired to these identifiers and take advantage of that fact to know which messages they are to monitor or subscribe to. (Uniqueness is not enforced in code but I guess it could be added by someone ambitious).

So the model holds arrays of both TestSets and Tests and a method to iterate through and perform all tests. So now we will look at the TestSets and then the Tests themselves. The model constructor is passed a parsed JSON object from which it builds all of the TestSet and Test object collections.

TestSets

There are currently five TestSets in the model described by the JSON file. These correspond to the five command buttons in the top-level GUI window and to the five lower-level GUI panels that display the actual tests.

The constructor for a TestSet is fairly simple. It saves its own name and prepares a dictionary to keep track of current state of tests it is watching and the Model constructor adds the new TestSet to its own TestSetArray.

Note

It does not hold references to the Tests themselves!!

Then for each TestSet, the model constructor creates the new Test objects based on the JSON. Each test is added to the model’s TestArray and is also passed to the appropriate TestSet via the AddWatchedTest() method so the TestSet can start to monitor and track test state via Notifications.

Note

The Test object reference itself is not retained in the TestSet, it is only used to establish the correct name and notification observers in the TestSetsTestsDictionary

TestSet instances need to watch for two types of notifications for each test. The test pass/fail “Results” and for any changes to the state of the Checkbox on the GUI for each test (will revisit this later). Suffice it to say at this point, TestSets register to receive messages of type testName.Results and testName.CheckboxState. Whenever it receives either one, its RecievedNotification method fires telling itself to revaluate the state of its internally held result set. Basically it needs to decide if the button on the main panel needs to be repainted red, yellow or green based on the roll-up of all the tests it is assigned to watch. If there is a change in outcome, then it publishes a notification of testSetName.Results.

Note

It uses testSetName, not testName

Tests

Each test object must have a unique name that identifies it. This is set by passing the JSON TestName to the Test constructor during the model initialization. Inside of the Test constructor the test also registers itself to receive notifications of type “Value” that match this name. (More on this later as we discuss Notifications.). Tests also have properties and methods supporting the type of test to execute.

A few more of those properties, TestType, minimum and maximum, missing value, are populated from the JSON file. TestType must be one of the enumerated TESTTYPES in the Test Class definition. The most commonly used are MIN_MAX and TF_PASS_T (read as true/falls pass true). Parameters of minimum, maximum and missingvalue only have meaning in the context of a MIN_MAX or MIN_MAX_MISSING type test.

Two important ‘computed properties’ on Test object deserver highlighting. (Xojo ‘computed properties’ are a special kind of property that can be assigned or read through their Get() and Set() methods.) See the TestValue.Set() and TestResults.Set() methods of the property. When a correctly matching named ‘Value’ notification is received, the ReceivedNotification method runs and assigns the received notification’s value to the TestValue property (calls its Set() function). That triggers the test to run its PerformTest() method. The pass/fail result is then assigned to the TestResult property. That properties Set() function then publishes a testName.”Results” message back out to any subscribers – for example back to the watching TestSet as described above.

Test Controls

Each test is represented on the GUI by a TestControl component on one of the five TestSet sub-panels. A TestControl is a Xojo compound control build with a TextField that display the value being tested, a button, the alarm acknowledge checkbox that toggles yellow or red states (and a ‘hidden’ component that allows resizing).

dp-ug3

It also has properties that it uses to maintain its own state view. TestControls both publish and receive Notifications.

  • The can publish checkbox state change notifications thru their check box Action() Xojo event. (testName.CheckboxState notification, see WatchedTestName property warning below!)
  • They can publish ‘ShowHelp” notifications thru their command button Action() Xojo event.
  • They are registered to receive both testName.Results and testName.Value notifications. This registration is done via the Xojo window Open() event for all TestControls on the each testset panel.

Warning

There are a couple important properties you need to watch out for…. (remember the messy bits I promised? Here are a couple really big ones…)

  1. The WatchedTestName property is set in the Xojo IDE at design time. This must match the name in the JSON file for this test (which also matches the TestName in the data model. It is used to create the notification message back to the data model TestSets for the CheckBoxState. ( the testName.CheckboxState notifications). I could have made the standard Xojo Checkbox action() event ‘reach out’ and call some method on TestSets, or could have created a Xojo custom ‘RaiseEvent’. However all of these have drawbacks eluded to earlier that require ‘reaching across’ model-view-controller boundaries. This way, for a tiny bit of developer-discipline, the TestControl is otherwise pretty well contained.
  2. The control’s Name property is set in the IDE at design time, when the control is positioned on the window panel. By convention only, its set to something that reflects the JSON file testName preceeded by ‘tc’ . Eg like tcShipGPSFixAge. This name is not used within in code, but it sure helps to have something consistent to keep track of where you’re working in the code.
  3. Each button Caption property is hard coded in the IDE at design time when the control is positioned on the window panel.

Warning

I'm going say it again in case you missed it… that WatchedTestName property on TestControls is really important for forming notification messages. It’s important to not confuse this property with WatchTestSetName described below.

Observer Color Buttons

Another custom control is the ObserverColorButton. You will find five of them on the main top-level window (winMain) – one for each TestSet panel. It is a subclass of the standard Xojo Bevel Button that holds additional properties of “Assigned Color” and WatchedTestSetName. It’s coded as a button subclass so it can implement the ReceivedNotification method of the MBS NotificationObserver class interface. As with TestControls, the WatchedTestSetName is set at design time in the IDE to match the TestSetName in the JSON file. don’t miss this point: this hard-coded indentifier association is important to allow the notifications work properly.

The winMainOpen() event includes a call to method InitializeObservers() which registers each ObserverColorButton button for the correct testSetName.Results notifications using the WatchedTestSetName (notification published from the TestSets own RecievedNotification() method).

The ReceivedNotification() method on the ObserverColorButton handles painting that top-level button’s Red, Yellow, Green background. In addition to managing the color, the button’s Action() event raises the TestSet’s detail panel (or window) that displays the detailed results within TestControls for the subset of tests.

Main Window Controls

winMain is the main window that is raised when the application starts. (In the Xojo IDE, selecting the App object in the project panel, will then show in the property inspector that winMain is the Default Window).

In addition to the five main ObserverColorButtons described above, winMain contains non-visual components (timers and sockets) that collectively form what I view as being the “Controller” in the logical MVC paradigm. The timers and sockets get initialized in the winMain.Open() event handler.

Where is the Contoller: Timers and Sockets

The primary notification system for test publishers, observers and their relationships are diagramed in Figure 2 below. Please refer to it as we work thru the discussion following.

dp-ug4

Figure 2 - Classes and Flow

There are currently six Xojo UDP sockets and three Xojo timer controls on winMain. Sockets and timers work together to obtain fresh data from the external servers. Each socket is associated with specific UDP ports during startup in the winMain.Open() event. When the timers periodically fire, their Action() event sends a UDP datagram request to the intended socket(s). Then it is the socket DataAvailable event handler that initiates a cascade of possible notifications to the data model and the graphical controls.

For the four sockets bound to ‘loggers’ (e.g. shipnav, video, rovctd and dataprobelogr ), the external server can provide two kinds of responses – either the comma-separated list of column names or the comma-separated data itself.

For example, if you examine the Timer1.Action() method you will see that after every sixty data payload requests, one is made for a HEADER record.

// every 60th time, request the header
If sockShipNav_Count Mod 60 = 0 Then 
  sockShipnav.Write Preferences.NavprocServerName, "HEADER"
  sockShipNav_Count = 0 //reset
Else 
  sockShipnav.Write Preferences.NavprocServerName, "Dataprobe"
End If
sockShipNav_Count = sockShipNav_Count +1

These four ‘logger’ sockets are the special ‘EnhancedSockets’ which are a subclass of the standard Xojo socket. This is so they can also hold a Xojo dictionary to store the header information which is used for parsing the normal data payloads.

When the external UDP server sends a response to a timer request, the corresponding sockets DataAvailable() event fires. (These are asynchronous events, and as this is UDP there is no guarantee either the requests or response will succeed).

If you examine the DataAvailable() methods for each of these you will see that the datagram payload is either processed into the dictionary for headers or else gets parsed into values - using the dictionary as an array index key. (This works because the first element in a logr file is always the LOGHIST.SYSTEM.UTC)

// if its a header datagram and the dictionary is 0 then we need to populate the dictionary
// can tell its header result if the first string token is "LOGHOST.SYSTEM.UTC"
if (   anArray(0).Compare("LOGHOST.SYSTEM.UTC") = 0) Then 

  If (me.HeaderIndexDictionary.Count =  0) then
    for i as integer = 0 to anArray.LastRowIndex
      me.HeaderIndexDictionary.Value(anArray(i)) = i
    next i
  end if  // else already did it once, just ignore
  return

End if

Within the DataAvailable() event method for a socket, each parsed element of interest (ie the tested item) has a Notification message formed of the pattern “TestName.Value” (eg. “ShipLatitude.Value” ) as shown below.

try
  idx = me.HeaderIndexDictionary.Value("SHIP.GPS.LAT")
  n.Name = "ShipLatitude"+".Value"
  n.Tag = anArray(idx)
  n.send(n)
catch e as KeyNotFoundException
end try

These notifications must be formed matching the TestName from the JSON file, because this “.Value” notification is what both the data model Test objects and the visual TestControl objects have subscribed to watch for. (in the Test data model object to execute the test and store state, and the TestControls to display the value being tested in its visual textbox.)

It is this sequence of timers making periodic socket requests, resulting in socket DataAvailable events which push out testname.Value notifications that further cascade to testname.Results notifications that drive the both data model and the GUI.

The notification cascade continues as the data model TestSet objects are receive the testname.Results notifications they registered for. This triggers the TestSet ReceiveNotification() method. The TestSet checks its internal state dictionary and if the Test notification results in a difference in the outcome of the ‘testset rollup’ (eg. green,yellow,red) it will publish a new Notification message formed of the pattern “TestSetName.Value” (ex. “ShipNav.Value” ). This is the notification type that one of the five top-level ObserverColorButton’s will have registered to watch for.

MISCELLANEOUS SOCKETS AND TIMERS DETAILS

A few miscellaneous timer/socket activites on winMain deserve mention here.

The sockNTP is a standard (not subclassed) Xojo socket that is used to get network time from the shipboard NTP server. It gets a request initiated at the bottom of the Timer1.Action and its DataAvailable() parses the payload and handles the notifications similar to above.

The sockBDF socket is a standard (not subclassed) Xojo socket that is used to get the logging host computer’s available disk space. It requires a perl “bdfserver” to be running on the host computer. (see /coredata/bin/bdfserver.pl)

Also near the end of the Timer1.Action() event handler, we grab the local host Date/Time and send a network ping out to the navproc . These actions support the NetworkPingAge and UnixClockDatetime tests “.Value” notifications.

DERIVED VALUE COMPUTERS

There are just a few instances where we need to compute derived values such as Depth, Salinity and PressureDifference to satisfy tests. We define a ‘Computer’ class for each of these who’s job it is to register as observers to Value notifications of their input parameters, and publish Value notification of their computed values back out (to start the cascade just as UDP sockets do above).

NOTE: these testName.Value observers and the published derived name are hard-coded in both the constructor and the ReceivedNotification() methods of the Computer

An instance of these computer classes are attached as properties of either the winRovCtd or winRovNav windows and are “new’d” into existance on the windows Open() event.

They are pretty simple classes. For example examine the Constructor and ReceivedNotification methods of the DepthComputer. The constructor registers as observer for Pressure and ShipLatitude .Value notifications. Then in ReceivedNotification() it computes Depth on new values and publishes the result as an “RovDepth.Value” notification.

HELP FILE SYSTEM

The Dataprobe Help system consists of a set of Help files located in a sub-directory of the Dataprobe Resources folder (alongside the .JSON file). A re-examinining of Figure 1 or the JSON file you will notice a “HelpFile” item with a relative path associated with each Test. When the data model was initialized from the JSON file, a Model property called HelpSystemDict (of type dictionary) was populated from the TestName and this file path.

Referring again to Figure 2. There is a Xojo Window object named winHelp that is opened and immediately hidden during the App.Open() event. The winHelp.Open() event holds an instance of HelpRequestObserver which is a sub-class of the NotificationObserverMBS class. This class receives ALL notifications. ReceivedNotification() event handler filters these for events with a name containing the string “ShowHelp”. When a user presses one of the TestControl buttons, its Action() event publishes a special notification formed from “ShowHelp” with a tag value (the payload) containing the TestName (eg. notification name = “ShowHelp”, tag = “ShipLatitude”). When the HelpRequestObserver detects any “ShowHelp” request it looks up the senders ‘TestName’ from the HelpSystemDict and populates the winHelp window with the appropriate help file text and makes the window visible.

THE MBS NOTIFICATION PLUGIN

A 3d-party Xojo plugin class purchased from Monkeybread Software (MBS) is the key component used to handle the notifications. It has multiple modes of operation and can be studied on its own in the demo program supplied by MBS (see whatever directory you have your MBS Plugins loaded to – in my case currently its my Home directory (rich/XOJO/MBS2020/MBS_XOJO_Plugins_V19.5/Examples/DataTypes/Notification.xojo_binary_project).

The MBS plugins must be licensed in order to compile the Dataprobe executable. The 64-bit version plugins must be installed into your Xojo plugins folder. ( my current case in Applications/Xojo 2019 Release3.1/Plugins folder)

The following MBS plugins are required:

  • MBS Xojo Main Plugin.xojo_plugin
  • MBS Xojo DataTypes Plugin.xojo_plugin

And your valid MBS license key must be properly registered in the App.CheckMBSLicence() method of the Xojo project (see App.EventHandlers.Open and CheckMBSLicense() in the Xojo IDE for the project) ** the license registration code is obscured on the version controlled project. You will need to substitute valid license keys obtained from your MBS email when you purchased the plugin.

NotificationObserverMBS facilities are used in two ways:

  • Classes and controls register to receive specific notifications from the NotificationMBS class. The NotificationMBS class handles distributing messages to observers through their ReceivedNotification events.
  • NotificationObserverMBS class can be sub-classed to receive ALL notifications. The developer then implements the ReceivedNotification event handler to act on the notifications needed. An example of this mode is the Help System described earlier. The HelpRequestObserver subclasses NotificationObserverMBS and its RecievedNotification() filters all notifications for “ShowHelp”.

APPENDIX 1: Preferences File

Xojo Dataprobe looks for its startup configuration in a set of ‘Preferences’ utilizing a PreferencesModule – a set of classes provided by Paul Lefebvre at Xojo. In a nutshell… in the App.Open() event, a JSON ‘Preferences’ file is read from a known location in the users Home folder. ( Note that this is to be confused with the JSON Tests configuration file mentioned elsewhere). That location may be different on Linux, OSX and Windows. A template file is created if one does not already exist in the users directory. That template file will then need to be edited.

  • OSX: /Users/xxxx/Dataprobe/Resources/Dataprobe.pref
  • Windows: C:\Users\xxxx\Dataprobe\Resources\Dataprobe.pref
  • Linux: /Users/xxxx/Dataprobe/Resources/Dataprobe.pref

An example of that file is shown below.

{

  "ROVName":"Ventana",
  "LogLevel":"NONE",
  "NavprocServerName":"rcnavproc1.rc.mbari.org",
  "TestsConfigFile":"VentanaDataprobeTests.JSON",
  "ShipnavPortNumber":54003,
  "VideoPortNumber":54005,
  "RovctdPortNumber":54007,
  "DataprobePortNumber":54009,
  "WinfrogPortNumber":54017,
  "BDFServer":"rcnavproc1.rc.mbari.org",
  "BDFPortNumber":54022,
  "NTPServer":"rcnavproc1.rc.mbari.org",
  "NTPPortNumber":123,
  "DefaultTestControlWidth":230,
  "DefaultTestControlHeight":38
}

These name:value pairs need to be configured for the platform (WesternFlyer or RachelCarson etc.). Note that the servers and port number need to match how the core data loggers etc. are configured. Also note the pair “TestsConfigFile”. This is the JSON file that define the Tests and TestSets described throughout this document. The tests and testset names must be kept in sync with the hard-coded names in various locations, as pointed out within this document.

Examine the PreferencesModule within the Dataprobe project tree using the IDE. Specifically examine the documentation found within “Notes” folder for the module which documents the Load and Save sequences for preferences.

For further reference, also see: https://blog.xojo.com/2014/01/27/saving-preferences/).

Appendix 2: Resizeable Windows and Controls

Xojo Dataprobe makes use of a 3d-party class called RubberViews to manage widow resize events to correctly resize all controls and fonts. I purchased a ‘lifetime unencrypted’ license from https://rubberviews.com/.

RubberViews classes and controls are installed by adding the provided RubberViews_Classes folder to the project folder. Then adding that folder into the Xojo IDE project space. Then the various classes become available as components in the Xojo IDE components Library.

Each graphical window or container control (including TestControls) to be resizeable must:

  • Have a RubberViews control added to it (selected from the controls library and dropped onto the component. (It displays as a small grey square in the ide – very hard to spot…) It will automatically receive a default name like ‘RubberViews1’
  • Each window or component’s Open() event, must initialize the control like ‘RubberViews1.Init(Self)’
  • Each window or component’s must also implement the Resized() and Resizing() event handlers.
  • In the case of our special colored buttons on testControls and main winMain, Resized() event may need to manually repaint the button.

APPENDIX 3: Additional Debugging Tools

In addition the standard debugging/ breakpoint tools provided by Xojo, Dataprobe has a couple of goodies that can be used.

BREAK ON EXCEPTIONS

This is more just a ‘tip’. As development of an app transitions from prototype to more of a ‘real’ app, your error handling starts to get under control with try/catch blocks etc., I generally turn off project ‘Break On Exceptions’ at some point and let my coded error handling function. (Menubar>Project>BreakOnExceptions) Then I only turn it back on when I need to step through to find some exception case I missed.

LOGS

Dataprobe has a logging capability built in. A ‘LogLevel’ of DEBUG, INFO, WARNING, FATAL or NONE is specified in startup JSON Preferences file. Timestamped logs will be found in the Logs directory specified there. The default LogLevel is NONE

Currently the only log message implemented is in App.Open() event handler. Look for the line: Utils.log4me(App.INFO,"Dataprobe log started.")

To create additional log entries, lines then can be added/enable in the code that call the Utils.log4me() method with the appropriate severity.

THE DEBUG NOTIFICATION WATCHER

Dataprobe has a window that can be enabled to watch for notifications. It is the winNotificationWatcher. It, combined with a NotificationObserverMBS subclass named FilteredNotificationObserver will allow you to filter for notifications that match a pattern.

It is commented-out in the project IDE. To enable it, go to App.Open() event and uncomment the line: //winNotificationWatcher.Show. Below is a screen grab of it in use. In this example we are filtering for both Value and Results notifications for the look like they contain the string “Lati”.So we are seeing latitude notifications for Ship and Rov.

dp-ug5

THE DEBUG NOTIFICATION PUBLISHER

Dataprobe has a window that can be enabled to publish notifications. It is the winNotificationValuePublisher. It allows you to simulate a testname.Value notification.

It is normally commented-out in the project IDE. To enable it, go to App.Open() event and uncomment the line: // winNotificationValuePublisher.Show. Below is a screen grab of it in use. In this example we are simulating a ShipLatitude value message. So when “Send” is pressed, its value of ‘456’ is sent out on the Notifications bus.

dp-ug6

Appendix 4 - Installation of the Application on Target Vessel

Installation on a target vessel should approximately mirror the default structure of the Development Project – minus the ‘src’ folder. After the project is built using the Xojo IDE Build icon. Xojo creates a deployable folder for each selected target operating system. (Currently OSX and Windows 64-bit). You will find them in subdirectories below Dataprobe/src/Builds-Dataprobe.

You will also need to inspect and edit the two files in the Resources folder – the Dataprobe.pref file and a xxxxTests.JSON for the target ROV system. These topics are covered in detail elsewhere in this document. Briefly though… Xojo Dataprobe looks for its startup configuration in a set of ‘Preferences’ utilizing a PreferencesModule – a set of classes written by Paul Lefebvre at Xojo. (see: “Appendix 1 – Preferences” of this document)

On the deployment machine you can delete the src directory if you don’t wish to support at-sea Xojo development environment.

To ‘deploy’ the app slightly varies on OSX or Windows. Basically you copy from the Builds folder executable to the top-level /Users/xxx/dataprobe folder depending on the target platform OS type. For OSX its simply the Dataprobe.app which is completely self-contained. For windows – you must drag the entire contents of the built ‘Dataprobe’ folder (ie. the .exe, dll’s and the Libs must all be kept together.

For example, on Windows your deployment for Ventana should look like:

/Users/xxxx/dataprobe
    /Dataprobe
        /Dataprobe Libs
            Dataprobe.exe
            (a bunch of dll files)
    /Resources
        Dataprobe.pref
        VentanaDataprobeTests.JSON
        /Help
            /RCARSON
                /Cameras
                /Network
                /RovCtd
                /RovNav
                /ShipNav

The pilots like a desktop shortcut to the dataprobe.executable.. (right-click on the exe and select ‘Create Shortcut”, then place it on the desktop. You should also change the icon for the shortcut to reflect the vessel. Right-click the shortcut and select ‘Properties’. Then from the pop-up, select ‘Change Icon”. Navigate to the correct ship .ico file in the Dataprobe/Resources folder.