MB-System Unix Manual Page: mbdatalist

### MB-System Unix Manual Page

#

##

##

##

##

##

##

##

##

##

##

    mbdatalist
    Section:  MB-System 5.0  (1)
    Updated:  3 June 2013
    Index

    NAME

    mbdatalist - parses recursive datalist files and outputs the
    complete list of data files, formats, and file weights.

    VERSION

    Version 5.0

    SYNOPSIS

    mbdatalist [-C -Fformat -Ifilename -N -O -P -Q -R -S -U -Y -Z -V -H]

    DESCRIPTION

    MBdatalist is a utility for parsing datalist files.
    Datalist files, or lists of swath data files and their format ids,
    are used by a number
    of MB-System programs. These lists may contain references
    to other datalists, making them recursive. See the MB-System
    manual page for details on the format and structure of datalists.
    The program mbdatalist outputs each
    swath data filename, format id, and file
    weight encountered as it descends through the input datalist tree. If a
    swath data file rather than a datalist is provided as input, the
    same swath data filename and format will be the sole output.

    This program
    can be used in shellscripts to read datalists in the same fashion
    as MB-System programs like mbgrid and mbprocess.
    This program can also be used to check and debug complex recursive
    datalist structures.

    The program mbprocess operates on "raw" swath data files,
    producing a "processed" swath data file (see the mbprocess man
    page for explanation).
    The MB-System algorithm for reading datalists will,
    if a flag is set, replace a swath file name with the associated
    "processed" file name when that "processed" file exists. This
    flag may be set by embedding "$PROCESSED" as a line in a datalist
    or it may be set first by the calling program. The flag may also
    be set to preclude reporting "processed" file names (embedding "$RAW"
    in a datalist accomplishes this). When setting this flag within
    datalists, the first encounter of a $PROCESSED or $RAW tag will
    prevail over later instances of either tag. The -P and -U
    options force mbdatalist to output processed file names
    when they exist (-P) or to only output unprocessed
    (raw) file names (-U).

    Programs such as mbgrid try to check statistics or "inf"
    files to see if the corresponding data files include data within
    the specified geographic bounds. Other programs look for
    "fast bathymetry" or "fast navigation" ("fbt" or "fnv") files
    in order to read the data more quickly. The -N option causes
    mbdatalist to create these three types of ancillary files
    for each swath data file. The -O option causes mbdatalist
    to create the "inf", "fbt", and "fnv" files only when they don't
    already exist or are out of date (older than the data file).

    Datalists may also contain a third value, called the grid
    weight, which is used by mbgrid to priortize data.
    The larger the grid weight, the more importance mbgrid
    attaches to the related bathymetry data. Grid weights can
    be applied to datalist entries which are themselves datalist
    files, causing these weights to be associated with all of
    files referenced therein. However, the default behavior is
    for any grid weight in a particular datalist entry to override
    values derived from higher levels in the recursive structure.
    This behavior can be reversed if a $NOLOCALWEIGHT tag is placed
    in the datalist, or in a datalist higher up in the structure.
    See the MB-System manual page for a more complete
    description.

    The -Rw/e/s/n option causes the program to check each
    data file with an "inf" file for overlap with the desired bounds,
    and only report those files with data in the desired area (or no
    "inf" file to check). This behavior mimics that of mbgrid,
    allowing users to check what data files will contribute to gridding
    some particular area.

    The -Q option causes the program to check each data file for
    the existence of any ancillary files (e.g. navigation files, edit
    save files, etc.) referenced in its mbprocess parameter file
    (if the parameter file exists). The program will list any problem
    found with the processing parameters, and will also list any
    data problem noted in the "inf" files. The possible data problems
    include:

            No survey data found

            Zero longitude or latitude in survey data

            Instantaneous speed exceeds 25 km/hr

            Average speed exceeds 25 km/hr

            Sounding depth exceeds 11000 m

            Unsupported Simrad datagram

    The -Z option causes the program to generate a datalist file
    named "datalistp.mb-1" and then exit. This datalist has the following form:

            $PROCESSED

            datalist.mb-1 -1

    This file is a commonly used convenience because it allows users to easily
    reference the swath files listed (directly or recursively) through
    the datalist "datalist.mb-1" with the $PROCESSED flag on. So, in order to
    grid the processed bathymetry rather than the raw bathymetry, run mbgrid
    with "datalistp.mb-1" as the input rather than "datalist.mb-1".

    The -S option causes mbdatalist to report the status of the files
    it lists, including whether the file is up to date or needs reprocessing,
    and if the file is locked. MBprocess sets locks while operating on
    a swath file to prevent other instances of mbprocess from simultaneously
    operating on that same file. This allows one to run mbprocess multiple
    times simultaneously on a single datalist, either on a single multiprocessor
    machine or on multiple computers mounting the same filesystem.
    The consists of creating a small text file
    named by appending ".lck" to the swath filename; while this file exists
    other programs will not modify the locked file. The locking program deletes
    the lock file when it is done. Orphaned lock files may be left if mbprocess
    crashes or is interrupted. These will prevent reprocessing by mbprocess,
    but can be both detected with the -S option and removed using the -Y option.

    Finally, this program can be used to copy the swath files referenced in a
    datalist structure to a single directory and to create a datalist there
    (names "datalist.mb-1") that references those swath files. This is
    accomplished using the -C option. The -C copy function will
    not be done if the -N, -O, or -Q options are specified,
    but is compatible with the -P, -R, and -U options.

    AUTHORSHIP

    David W. Caress (caress@mbari.org)

      Monterey Bay Aquarium Research Institute

    Dale N. Chayes (dale@ldeo.columbia.edu)

      Lamont-Doherty Earth Observatory

    OPTIONS

    -C

    Causes the swath files referenced in the input datalist structure to be
    copied to the current directory and creates a datalist
    (names "datalist.mb-1") that references the copied swath files.
    The copy function will not be done if the -N, -O, or -Q options are specified.
    If the -P, -R, and -U options are specified these
    functions will modify which swath files are copied. Any ancilliary files
    (e.g. *inf metadata files) will also be copied, but processed data
    files derived from the target copied files will not be copied.
    -F

    format

    Sets the data format associated with the datalist or swath data
    file specified with the -I option. By default, this program
    will attempt to determine the format from the input file suffix
    (e.g. a file ending in .mb57 has a format id of 57, and a file
    ending in .mb-1 has a format id of -1). A datalist has a format id
    of -1.
    -H

    This "help" flag cause the program to print out a description
    of its operation and then exit immediately.
    -I

    filename

    Sets the input filename. If format > 0 (set with the
    -f option) then the swath data filename specified by infile
    is output along with its format and a file weight of 1.0.
    If format < 0, then infile
    is treated as a datalist file containing a list of the input swath sonar
    data files to be processed and their formats.  The program will parse
    the datalist (recursively, if necessary) and output each swath filename
    and the associated format and file weight.
    -N

    This argument causes MBdatalist to generate three types of ancillary
    data files ("inf", "fbt", and "fnv").
    In all cases,
    the ancillary filenames are just the original filename with
    ".inf", ".fbt", or ".fnv" appended on the end.
    MB-System makes use of ancillary data files in a number
    of instances. The most prominent ancillary files are metadata or
    "inf" files (created from the output of mbinfo).
    Programs such as mbgrid and mbm_plot try to check "inf"
    files to see if the corresponding data files include data within
    desired areas. Additional ancillary files are used to speed
    plotting and gridding functions. The "fast bath" or "fbt" files
    are generated by copying the swath bathymetry to a sparse,
    quickly read format (format 71). The "fast nav" or "fnv" files
    are just ASCII lists of navigation generated using mblist
    with a -OtMXYHSc option. Programs such as mbgrid,
    mbswath, and mbcontour will try to read "fbt" and "fnv" files
    instead of the full data files whenever only bathymetry or
    navigation information are required.
    -O

    This argument causes MBdatalist to generate the three ancillary
    data files ("inf", "fbt", and "fnv") if
    these files don't already exist or are out of date.
    -P

    Normally, mbdatalist allows $PROCESSED and $RAW tags within
    the datalist files to determine whether processed file names are
    reported when available ($PROCESSED) or only raw file names are
    reported ($RAW). The -P option forces mbdatalist
    to output processed file names when they exist.
    -Q

    This option causes the program to check each
    data file for the existence of any ancillary files
    referenced in its mbprocess parameter file
    (if the parameter file exists). The relevant ancillary
    files include edit save files generated by mbedit
    or mbclean, navigation files generated by mbnavedit
    or mbnavadjust, tide files, and svp files. An error message
    is output for each missing ancillary file.
    -R

    w/e/s/n

    The bounds of the desired area are set in longitude
    and latitude using w=west, e=east, s=south, and n=north.
    This option causes the program to check each
    data file with an "inf" file for overlap with
    the desired bounds, and only report those
    files with data in the desired area (or no
    "inf" file to check). This behavior mimics that of
    mbgrid, allowing users to check what data files will
    contribute to gridding some particular area.
    -S

    This option causes mbdatalist to report the status of the files
    it lists, including whether the file is up to date or needs reprocessing,
    and if the file is locked. MBprocess sets locks while operating on
    a swath file to prevent other instances of mbprocess from simultaneously
    operating on that same file. Locking consists of creating a small text file
    named by appending ".lck" to the swath filename; while this file exists
    other programs will not modify the locked file. The locking program deletes
    the lock file when it is done. Orphaned lock files may be left if mbprocess
    crashes or is interrupted. These will prevent reprocessing by mbprocess,
    but can be both detected and removed using mbdatalist.
    -U

    Normally, mbdatalist allows $PROCESSED and $RAW tags within
    the datalist files to determine whether processed file names are
    reported when available ($PROCESSED) or
    only (raw) unprocessed file names are
    reported ($RAW). The -U option forces mbdatalist
    to only output raw file names.
    -V

    Normally, mbdatalist only prints out the filenames and formats.
    If the -V flag is given, then mbinfo works in a "verbose" mode and
    outputs the program version being used.
    -Y

    This option causes mbdatalist to remove any processing locks on files
    it parses. MBprocess and other programs may set locks while
    operating on a swath file to prevent other programs from simultaneously
    operating on that same file.The consists of creating a small text file
    named by appending ".lck" to the swath filename; while this file exists
    other programs will not modify the locked file. The locking program deletes
    the lock file when it is done. Orphaned lock files may be left if MB-System
    programs crash or are interrupted. These can be detected using the -S
    option of mbdatalist.
    -Z

    The -Z option causes the program to generate a datalist file that
    will first set a $PROCESSED flag and then reference the input
    file specified using the -Ifilename option.
    The output datalist is named
    by adding a "p.mb-1" suffix to the root of the input file (the root is
    the portion before any MB-System suffix).

    By default, the input is assumed to be a datalist named datalist.mb-1, resulting
    in an output datalist named datalistp.mb-1 with the following contents:

            $PROCESSED

            datalist.mb-1 -1

    If the input file is specified as a datalist like
    datalist_sslo.mb-1, then the output datalist
    datalist_sslop.mb-1
    will have the following contents:

            $PROCESSED

            datalist_sslo.mb-1 -1

    If the input file is specified as a swath file like
    20050916122920.mb57, then the output datalist 20050916122920p.mb-1
    will have the following contents:

            $PROCESSED

            20050916122920.mb57 57

    EXAMPLES

    Suppose we have two swath data files from an EM3000 multibeam
    and another two from an Hydrosweep MD multibeam. We might
    construct two datalist files. For the EM3000 we might
    have a file dlst_em3000.mb-1 containing:

            0004_20010705_165004_raw.mb57 57

            0005_20010705_172010_raw.mb57 57

    For the Hydrosweep MD data we might have a file dlst_hsmd.mb-1
    containing:

            al10107051649.mb102 102

            al10107051719.mb102 102

    Further suppose that we have found it necessary to edit
    the bathymetry in 0005_20010705_172010_raw.mb57 and
    al10107051719.mb102 using mbedit, and that mbprocess
    has been run on both files to generate processed files
    called 0005_20010705_172010_rawp.mb57 and
    al10107051719p.mb102.

    If we run:

            mbdatalist -I dlst_em3000.mb-1

    the output is:

            0004_20010705_165004_raw.mb57 57 1.000000

            0005_20010705_172010_raw.mb57 57 1.000000

    Here the file name is followed by the format and then
    by a third column containing the default file weight of 1.0.

    Similarly, if we run:

            mbdatalist -I dlst_hsmd.mb-1

    the output is:

            al10107051649.mb102 102 1.000000

            al10107051719.mb102 102 1.000000

    If we insert a line

            $PROCESSED

    at the top of both dlst_hsmd.mb-1 and dlst_em3000.mb-1,
    then the output of mbdatalist changes so that:

            mbdatalist -I dlst_em3000.mb-1

    yields:

            0004_20010705_165004_raw.mb57 57 1.000000

            0005_20010705_172010_rawp.mb57 57 1.000000

    and:

            mbdatalist -I dlst_hsmd.mb-1

    yields:

            al10107051649.mb102 102 1.000000

            al10107051719p.mb102 102 1.000000

    Now suppose we create a datalist file called dlst_all.mb-1
    that refers to the two datalists shown above (without the
    $PROCESSED tags). If the contents of dlst_all.mb-1 are:

            dlst_em3000.mb-1 -1 100.0

            dlst_hsmd.mb-1   -1   1.0

    where we have specified different file weights for the
    two datalists, then:

            mbdatalist -I dlst_all.mb-1

    yields:

            0004_20010705_165004_raw.mb57 57 100.000000

            0005_20010705_172010_raw.mb57 57 100.000000

            al10107051649.mb102 102 1.000000

            al10107051719.mb102 102 1.000000

    Now, if we use the -P option to force mbdatalist
    to output processed data file names when possible, then:

            mbdatalist -I dlst_all.mb-1 -P

    yields:

            0004_20010705_165004_raw.mb57 57 100.000000

            0005_20010705_172010_rawp.mb57 57 100.000000

            al10107051649.mb102 102 1.000000

            al10107051719p.mb102 102 1.000000

    SEE ALSO

    mbsystem(1)

    BUGS

    No true bugs here, only distantly related arthropods... Yum.
    Seriously, it would be better if the copy function preserved
    the modification times of the copied swath files and ancilliary
    files. Copying of processed files should also be an option.

     Index

    NAME
    VERSION
    SYNOPSIS
    DESCRIPTION
    AUTHORSHIP
    OPTIONS
    EXAMPLES
    SEE ALSO
    BUGS

Last Updated: 3 June 2013

Return to list of MB-System manual pages...

Back to MB-System Home Page...