Migrating XML scripts to TethysL¶
We are working on continuing the migration of XML scripts to TethysL. The instructions in this page are intended to help with this process.
The conversion of a mission script involves three main steps:
- Automated conversion of the script from XML to TethysL
- Compilation and adjustments to the generated TethysL file
- Pushing the new TethysL script to the lrauv-mission repository
To perform these steps you can use the TethysL CLI or, recommended, VS Code (or a combination of both).
Here's a screencast of a VS Code session covering some initial steps with an XML file that happens to expose a couple of easy-to-fix structural issues.
In more detail, with
Maintenance/DUSBL.xml as an example, the steps are as follows:
The following steps assume that the
lrauv-mission repository (or a fork)
has been cloned and that the
lrauv-mission directory is the current working directory.
1. Automated conversion from XML to TethysL¶
Use TethysL CLI's
xml2tethysl command to convert the XML file into TethysL:
tethysl xml2tethysl Maintenance/DUSBL.xml
Unless a given
some/mission.xml is pretty badly constructed, the
some/mission.xml.tl, in our example:
.xml.tl file extension at this point
simply indicates the TethysL version has been automatically generated from the
We will be renaming this file to a simple
.tl extension in a later step.
The XML parser in TethysL is pretty lenient: it basically performs a direct transliteration of the given XML elements into corresponding TethysL constructs.
Two key aspects to keep in mind:
Depending on how compliant the given XML file is to the Tethys XML Schema, our subsequent compile step below on the generated TethysL script may expose issues with the original XML, for example, involving missing or out-of-order constructs, and other structural or syntax errors.
Pay special attention to the translation of expressions involving binary operations (
<Mult>, etc.) to verify the intended execution semantics is preserved. As of TethysL 2.9.0, this should be taken care of by the translator but we kindly ask you to double check. As a reminder, the LRAUV performs a flat-precedence, left-to-right evaluation of expressions involving binary operators, while TethysL follows the standard in most common programming languages (C/C++, Python, Java, etc.) regarding operator precedence and associativity. (You can see some related discussion here.) As needed or convenient, you can of course adjust any parenthesization on the resulting TethysL source toward a legible and less error-prone expression.
2. Refine generated
compile command to compile the TethysL version:
tethysl compile Maintenance/DUSBL.xml.tl
By design, the TethysL compiler is very strict and more comprehensive.
If errors persist, don't hesitate to ask for help if you are not sure how to fix them.
A successful compile of a
*.tl file not only validates the script,
but also generates a corresponding XML file, which is named by setting
.tx as the file extension.
In our example, the generated XML file is
.txextension was designated to denote "TethysL Compiled XML" files, so one can easily tell them apart from the regular files originally authored in XML.
.txfiles are not to be pushed to the repository.
- In VS Code, the generation of the
.txis not enabled by default, but there is a command to toggle the corresponding flag.
Using a diff tool or by doing a direct side-by-side inspection,
compare the generated
DUSBL.tx with the original
to verify they both represent the same logical mission or aggregate,
that is, keeping in mind they may be formatted differently.
We have added VS Code commands to help with this step. See below.
Any difference in the namespace declarations?
You can disregard any differences in the namespace declarations:
- In general, they are not well maintained in the original XML files
- TethysL automatically adds the correct namespace declarations (and only the ones actually needed)
You can complete this step by running the
prettify CLI command
(or using the corresponding command in VS Code)
to format the TethysL file:
tethysl prettify Maintenance/DUSBL.xml.tl
3. Update lrauv-mission¶
At this point, for the target mission that has been converted, you will have these three files, with only the original XML currently committed in the git repository:
||Original XML script|
||TethysL conversion, including your adjustments (if any)|
||"Compiled XML," as by-product of compiling the TethysL|
Once you are happy with the resulting
the final steps are to
rename it, add it to the git repository, and remove the original XML file.
More concretely, assuming a pull-request workflow and a new branch in the 'origin' remote for purposes of the PR:
- Rename the TethysL script to remove the
mv Maintenance/DUSBL.xml.tl Maintenance/DUSBL.tl
- Create a branch and check it out, e.g.:
git checkout -b xml2tl/Maintenance/DUSBL
- Add the TethysL script to the git repository:
git add Maintenance/DUSBL.tl
- Remove the XML script from the git repository:
git rm Maintenance/DUSBL.xml
- You can remove the compiled XML file (no longer needed):
- Commit and push:
git commit -m "Convert Maintenance/DUSBL to TethysL" git push origin xml2tl/Maintenance/DUSBL
- Create pull request in the remote repository.
VS Code Commands¶
As mentioned above, the TethysL extension for VS Code includes commands intended to facilitate the migration from XML to TethysL. They are described next.
TethysL: Generate TethysL (.xml.tl) from this XML
.xml open in the editor, this command will generate a corresponding
xml/tx Diff (1)¶
TethysL: Pretty-print both xml/tx and open compare window
This command relies on the
xmldiff python tool,
which is assumed available via the
PATH on your machine.
xmldiff has been installed via
pip install --user xmldiff.
Make sure you have the
.tx generation enabled.
Enabled for any of the three related files
this command performs these steps:
- Re-formats both the
.txfiles (using the
--pretty-printoption provided by
xmldiff), as a way of "normalizing" the files in terms of whitespace prior to comparison.
- Opens a VS Code Diff window comparing the two formatted files for your visual inspection.
The result looks like this:
xml/tx Diff (2)¶
TethysL: Diff xml/tx using xmldiff
This command performs a direct
execution on the original
.xml and the generated
The output of then is shown in a VS Code panel.
Prettify XML (1)¶
TethysL: Prettify XML using xmldiff --pretty-print
This command runs
xmldiff --pretty-print to
re-format the currently open
Prettify XML (2)¶
TethysL: Prettify XML using prettyMission.py
This command requires the
lrauv-application codebase available on your machine to be functional.
With the lrauv-application codebase available on the system, you can directly
Tools/xmlformat/prettyMission.py on your
.tx mission from within VS Code.
While the TethysL compiler is pretty comprehensive in terms of syntax and semantic validation, unless you can verify exact logical equivalence of the generated
.txand the original
.xmlin the procedure above, there might be migration cases where the resulting TethysL script should also be tested against an LRAUV simulator or even a real vehicle. Please check with an LRAUV expert about such potential cases, probably as part of the description of your pull request.
If your original XML contains
<Insert>'s referring to
.xmlfiles, that's fine; both XML and TethysL
insert's are allowed so there is no need to first convert those.
TethysL offers enhanced capabilities at the syntax level, including arrays and macros, which make it possible to express some mission scripts in a more concise way (in some cases, very significantly). Although the automated conversion from XML to TethysL (first step above) will not make use of these features, you can consider using them in some future TethysL version of the script if they are applicable.
Ideally, we would like the original authors or recent committers of the XML files to be directly involved in the respective migrations, such that authorship can at least be preserved to some extent. Note that the git history of a converted script will be lost1 due to the file renaming involved.
There are possible ways (including git-filter-repo) that we could consider using to better preserve history and authorship, but this aspect has not been discussed with the LRAUV team, and is not currently planned. ↩