Skip to content

Migrating XML scripts to TethysL

We are working on continuing the migration of XML scripts to TethysL.

Ideally, we would like the original authors, or recent committers of the XML files, to take care of the respective conversions, such that authorship can be preserved, at least to some extent. These instructions are intended to help with this process.

The conversion of a mission script involves three main steps:

  1. Automated conversion of the script from XML to TethysL
  2. Compilation and adjustments to the generated TethysL file
  3. Pushing the new TethysL script to the lrauv-mission repository

Only requirement is the TethysL CLI on your computer. You can also use VSCode.

Here's a screencast of a VSCode session covering some initial steps with an XML file that happens to expose a couple of easy-to-fix structural issues.

In more detail, with Maintenance/DUSBL.xml as an example, the steps are as follows:

The following steps assume that the lrauv-mission repository (or a fork) has been cloned and that the lrauv-mission directory is the current working directory.

1. Automated conversion from XML to TethysL

Use TethysL CLI's xml2tethysl command to convert the XML file into TethysL:

tethysl xml2tethysl Maintenance/DUSBL.xml

Unless a given some/mission.xml is pretty badly constructed, the xml2tethysl command should generate some/mission.xml.tl, in our example: Maintenance/DUSBL.xml.tl.

The .xml.tl file extension at this point simply indicates the TethysL version has been automatically generated from the .xml. We will be renaming this file to a simple .tl extension in a next step.

Important

The XML parser in TethysL is pretty lenient and rather simplistic: it basically only performs a direct, blind transliteration of the given XML elements into corresponding TethysL constructs.

Two key aspects to keep in mind:

  • Depending on how compliant the given XML file is to the Tethys Schemas, our subsequent compile step below on the generated TethysL script may expose issues with the original XML, for example, missing or out-of-order constructs, and other structural or syntax errors.

  • Expressions involving binary operations like <Add>, <Sub>, <Mult>, <Or>, And, <Ge>, <Lt>, etc., may not get properly reflected on the TethysL side in terms of the intended precedence and associativity of the operators. Please review and adjust the resulting expression as needed on the TethysL side. Note that TethysL follows a more standard practice in these regards, so it should be more intuitive and less error-prone to indicate the intended grouping here as needed. (You can see some related discussion here.)

2. Refine generated .xml.tl

2.1. Compile

Use CLI's compile command to compile the TethysL version:

tethysl compile Maintenance/DUSBL.xml.tl
Fix any reported errors in the TethysL version, and recompile it until successful.

By design, the TethysL compiler is very strict and more comprehensive.
If errors persist, don't hesitate to ask for help if you are not sure how to fix them.

A successful compile of a *.tl file not only validates the script, but also generates a corresponding XML file, which is named by setting .tx as the file extension. In our example, the generated XML file is Maintenance/DUSBL.tx.

.tx ?

  • The .tx extension was designated to denote "TethysL Compiled XML" files, so one can easily tell them apart from the regular files originally authored in XML.
  • The .tx files are not to be pushed to the repository.
  • In VSCode, the generation of the .tx is not enabled by default, but there is a command to toggle the corresponding flag.

2.2. Compare

Using a diff tool or by doing a direct side-by-side inspection, compare the generated DUSBL.tx with the original DUSBL.xml to verify they both represent the same logical mission or aggregate, that is, keeping in mind they may be formatted differently.

There are XML Extensions for VSCode that facilitate this comparison. You can also consider a tool like xmldiff, which makes a good logical comparison based on the XML structure. One other option, if you also have the lrauv-application codebase on your system, is to run Tools/xmlformat/prettyMissions.py on both files (as a way of "normalizing" them in terms of whitespace), and then run a more regular diff on the resulting formatted files.

Disregard any differences in the namespace declarations

  • In general, they are not well maintained in the original XML files
  • TethysL automatically adds the correct namespace declarations (and only the ones actually needed)

You can complete this step by running the prettify CLI command (or using the corresponding command in VSCode) to format the TethysL file:

tethysl prettify Maintenance/DUSBL.xml.tl

3. Update lrauv-mission

At this point, for the target mission that has been converted, you will locally have these three files, with only the original XML currently committed in the git repository:

File Description
Maintenance/DUSBL.xml Original XML script
Maintenance/DUSBL.xml.tl TethysL conversion, including your adjustments (if any)
Maintenance/DUSBL.tx "Compiled XML," as by-product of compiling the TethysL

Once you are happy with the resulting Maintenance/DUSBL.xml.tl, the final steps are to rename it, add it to the git repository, and remove the original XML file.

More concretely, assuming a pull-request workflow and a new branch in the 'origin' remote for purposes of the PR:

  • Rename the TethysL script to remove the .xml part:
    mv Maintenance/DUSBL.xml.tl Maintenance/DUSBL.tl
    
  • Create a branch and check it out, e.g.:
    git checkout -b xml2tl/Maintenance/DUSBL
    
  • Add the TethysL script to the git repository:
    git add Maintenance/DUSBL.tl
    
  • Remove the XML script from the git repository:
    git rm Maintenance/DUSBL.xml
    
  • You can remove the compiled XML file (no longer needed):
    rm Maintenance/DUSBL.tx
    
  • Commit and push:
    git commit -m "Convert Maintenance/DUSBL to TethysL"
    git push origin xml2tl/Maintenance/DUSBL
    
  • Create pull request in the remote repository.

Additional notes

  • While the TethysL compiler is pretty comprehensive in terms of syntax and semantic validation, unless you can verify exact logical equivalence of the generated .tx and the original .xml in the procedure above, there might be migration cases where the resulting TethysL script should also be tested against an LRAUV simulator or even a real vehicle. Please check with an LRAUV expert about such potential cases, probably as part of the description of your pull request.

  • If your original XML contains <Insert>'s referring to .xml files, that's fine; both XML and TethysL insert's are allowed so there is no need to first convert those.

  • TethysL offers enhanced capabilities at the syntax level, including arrays and macros, which make it possible to express some mission scripts in a more concise way (in some cases, very significantly). Although the automated conversion from XML to TethysL (first step above) will not make use of these features, you can consider using them in some future TethysL version of the script if they are applicable.

  • As mentioned, we would like the original authors or recent committers of the XML files to be directly involved in the respective migrations, such that at least authorship can be preserved to some extent. For a converted script, note that its git history per se will be lost1 due to the file renaming involved.


  1. There are possible ways (including git-filter-repo) that we could consider using to better preserve history and authorship, but this aspect has not been discussed with the LRAUV team, and is not currently planned.