<refentry id="intro" revision="17 Jan 2008">
<refmeta>
    <refentrytitle>Introduction to LCM</refentrytitle>
</refmeta>

<refnamediv>
    <refname>Introduction to LCM</refname>
    <refpurpose>An overview of how LCM works</refpurpose>
</refnamediv>

<refsect1>
    <title>What is LCM?</title>

    <para>LCM stands for Lightweight Communications and Marshalling.  The
    <emphasis>communications</emphasis> aspect of LCM is that it allows
    multiple senders and receivers to use a network for sending and
    receiving messages to each other using a shared channel.  The
    <emphasis>marshalling</emphasis> aspect of LCM is that it provides
    you the ability to define your own data types that are
    automatically encoded into data packets in a platform- and
    language-independent fashion.  Most users will use both of these
    capabilities at the same time even though they can be separated if
    desired. LCM also includes a logging facility, allowing LCM
    traffic to be recorded and replayed.</para>

    <para>The design goals for the communications system are:</para>
    
    <itemizedlist>
        <listitem>Supports many-to-many operation in which there may be several
        senders and receivers for a particular message type</listitem>
        <listitem>No per-network or per-host daemon needed for relaying
        of data</listitem>
        <listitem>Each packet transmitted appears on the wire no more than
        one time</listitem>
        <listitem>Minimal latency</listitem>
        <listitem>Scales to high bandwidth</listitem>
        <listitem>No practical limit on packet size</listitem>
    </itemizedlist>

    <para>The design goals for the marshalling system are:</para>

    <itemizedlist>
        <listitem>Absolute type-safety, in which an application that decodes
        a packet is certain that it was sent with the same type in mind,
        even if the two applications were compiled separately</listitem>
        <listitem>Platform- and language-independence, with the goal that
        at least C and Java would be first-class users</listitem>
        <listitem>Ease of use, so that data types can be manipulated using
        standard constructs of the target language both before and after
        marshalling (e.g. structs in C)</listitem>
    </itemizedlist>

    <para>These are some properties that LCM <emphasis>does not</emphasis>
    have:</para>

    <itemizedlist>
        <listitem>LCM does not guarantee delivery of a particular message.</listitem>
        <listitem>LCM does not guarantee ordering of messages.</listitem>
        <listitem>LCM is not easy to use across wide-area networks.</listitem>
    </itemizedlist>
</refsect1>

<refsect1>
    <title>Communications System</title>

    <para>LCM uses <emphasis>UDP multicast</emphasis> for its underlying
    communications mechanism.  The primary reason for this is so that
    multiple senders and receivers can connect to the same data channel
    without requiring a central server of some type that copies packets
    or maintains separate point-to-point links.  Instead, this maintainance
    is performed directly by the network stacks of the hosts, switches,
    and routers on the network.</para>

    <para>Another advantage of UDP multicast is that multiple applications
    on the same host behave much the same as multiple applications spread
    across multiple hosts.  This is in contrast to UDP broadcast, in which
    generally only one application per host can listen on a specific
    port.  UDP multicast also has the advantage of routability across multiple
    subnets as long as the routers are configured properly (generally
    possible within an organization but not Internet-wide).</para>

    <para>That said, UDP multicast can have quirks in getting it to work
    properly on a LAN, or even a single host.  These issues are discussed
    in the Section <xref linkend="multicast-setup"/>.</para>

    <!--
    <para>I THINK WE'RE GOING INTO TOO MUCH DETAIL HERE- SOME OF THIS
    SHOULD BE MOVED TO A TROUBLESHOOTING/CONFIGURATION SECTION. IT
    DOESN'T FEEL LIKE INTRODUCTORY MATERIAL.</para>
    -->

    <para>Using LCM requires you to pick an IP address to serve as the
    multicast group for all messages.  This IP address is distinct from
    any one host's address and is used by every machine to subscribe to
    the group.  You must also pick a UDP port over which messages will be
    exchanged.  The default IP is 239.255.76.67 and port is 7667.  This IP
    is in the special administrative range that is not routable between
    networks.  You must also choose a time-to-live (TTL) which represents
    the maximum number of routers that a packet can pass through.  The
    default of 0 prevents your packets from leaving a single host.  Using
    1 will allow your packets to be received anywhere on a single subnet.
    Note that multicast traffic is treated like broadcast traffic within
    a subnet, in that every machine on the subnet will receive the traffic,
    even if it is not subscribed to the multicast group.  This can be
    prevented using a feature called <emphasis>IGMP snooping</emphasis>
    which is available on higher-end managed switches.  See Section
    <xref linkend="multicast-setup"/>.</para>

    <para>After choosing the multicast group, LCM subscribe to the multicast
    group during initialization of the library by sending out the necessary
    IGMP requests onto the subnet.  After this subscription, that host will
    receive packets destined to the IP address assigned to the group.  The
    network stack of the host will assure that any applications listening on
    the correct UDP port receive the traffic.  Unlike normal UDP or TCP
    connections, multiple applications on the same host are allowed to listen
    to the same multicast port.</para>

    <para>The format of LCM packets is very simple.  A packet consists of a
    variable-length ASCII string identifier followed by variable-length buffer
    of opaque data.  This opaque data will generally be a marshalled LCM data
    type.  The string identifier is called the <emphasis>channel
    name</emphasis> and is the basis for the subscription mechanism in LCM.
    At the LCM library level, packets are filtered out that do not contain
    channel names to which the application has subscribed.  The packets
    corresponding to subscribed channels are passed through to the
    application, either with the opaque buffer intact or already unmarshalled,
    depending on which API is being used.</para>

    <para>Since UDP has a packet-size limit of 65K, LCM will automatically
    perform high-level fragmentation and reassembly if a packet requires
    it.</para>

    <para>Choosing UDP as the transport mechanism for LCM has some
    implications for the practical use of the library.  Most importantly,
    since packets could be lost, it is important to design your applications
    and protocols so this is not a problem.  It is advisable to transmit
    time-sensitive data at a fixed rate so that if a packet is lost, the
    receiving application will immediately recover with the most recent data
    as it is received.  The frequency of your transmit rate will be determined
    by how sensitive your application is to latency.  Transmitting at a fixed
    rate also has the advantage that your network will be operating under a
    steady-state amount of bandwidth and you will not need to worry about the
    effects of momentary spikes in traffic.</para>

    <para>Protocols requiring guaranteed request-response semantics can also
    be a challenge when using UDP.  It is not practical to implement such a
    scheme for a many-to-many network, however, if you need one-to-one
    communications as part of your LCM network, guaranteed delivery of
    request-response protocols can be achieved as follows: Encode the state of
    the "handshake" using data fields in both the request and response data
    types.  For example, the requester continues to send the request packet
    until the response packet contains confirmation of its receipt.  The
    responder then continues to resend the response until the requestor
    confirms receipt of it.  If your protocols are already designed around
    fixed data rates, this handshaking procedure is easy to implement.</para>
    
</refsect1> 

<refsect1>
    <title>Marshalling System</title>

    <para>Sending messages from one application to another involves
    encoding and decoding data into a byte stream. This sort of
    programming tends to be straight-forward at first glance, but it
    is tedious, error-prone, and intolerant to mismatches in types. A
    mismatch in type can occur in two fundamental ways: a message of
    type "foo_t" was expected but a message of "bar_t" was received,
    or a message of type "foo_t" was expected but the underlying type
    specification of "foo_t" did not match on the transmitter and
    receiver.</para>

    <para>Like other message specification systems, it addresses the
    tedious and error-prone aspects via automatic-generation of code,
    but provides first-class support for several languages (currently
    C, Java, and Python). Unlike other message specification systems,
    LCM detects both types of faults described above, preventing
    erratic (and often hard-to-isolate) failures. LCM also enforces
    safe use of variable-length arrays, helping to prevent illegal
    pointer operations in languages like C.</para>

    <para>The formal type specification language for LCM follows
    standard C and Java syntax as much as possible, while providing
    several additional features. The syntax itself will also be
    familiar to users of External Data Representation (XDR), used by
    RPC. Notable deviations include a simplified notation for
    variable-length arrays, and a consistent naming scheme for
    primitive types (based on C99's stdint).</para>

    <para>The tutorials illustrate all of the basic syntax and
    capabilities of the LCM marshalling system.</para>
</refsect1>

<refsect1>
  <title>Logging</title>

  <para>LCM includes a logging facility allowing LCM network traffic
  to be recorded and played back again. LCM's multicast-based network
  architecture means that adding a logger to the system requires no
  additional configuration: simply run the logger or log player. The
  result of a played-back log is indistinguishable from live network,
  aside from any timestamps that might be embedded in the user-defined
  LCM types. </para>

  <para>The log file format is very simple: each LCM message is
  concatentated in the order they are received from the network
  stack. For each record, a synchronization sequence is written,
  followed by the time at which the message was received, followed by
  the packet data. The timestamps allow the log player to closely
  replicate the inter-message timing that occured when the log was
  created. The synchronization sequence is used to facilitate seeking
  in the log file: the Java-based log player, which features a
  flexible GUI, allows the both seeking and variable-speed
  playback.</para>

  <para>The log file format itself has both C and Java implementations
  and can be accessed by users. This is useful for off-line processing
  of log files: messages can be filtered or otherwise altered. A
  similar result could be obtained by writing an LCM application and
  playing the original log while recording a new log, however
  accessing the log files directly is both faster (since the log file
  can be processed at the maximum speed your application is capable
  of) and more accurate (since timestamps in the logfile can be
  preserved exactly).</para>

  <!--
  <para>SHOULD WE FACTOR THIS, AND OTHER PRACTICAL ADVICE, INTO A "HOW
  TO WRITE A WELL-BEHAVED LCM APPLICATION" SECTION? IT COULD
  INCLUDE NAMING CONVENTIONS.</para>.
  -->

  <para>Allowing applications to operate correctly from logged data
  requires some careful consideration of timestamps. Specifically,
  timestamps in logged data will not agree with the computer's current
  time. However, it is relatively easy to write well-behaved LCM
  applications that are time-agnostic.</para>

  <itemizedlist>
    <listitem>Every user-defined LCM type should include a timestamp,
      defined as "int64_t utime": the number of microseconds since the
      UNIX epoch, UMT. Messages arising from a physically-grounded event
      (e.g., a GPS reading or laser scan) should have this field
      initialized with the best estimate of the time at which that
      physical event occurred. </listitem>
    
    <listitem>Events that are not physically-grounded (i.e., the result
      of filtering a GPS measurement, or command to steer around an
      obstacle detected by a laser scan) should bear a timestamp that is
      derived from the physically-grounded event that caused it. In
      other words, these events should NOT use the computer's time: they
      should use their best guess of the local time as indicated by the
      physically-grounded events that are occuring. Adherence to this
      principle allows modules to operate on logged data, outputing
      timestamps that agree with the logged data.</listitem>

    <listitem>Supporting seeking in log files can require some
      additional logic in LCM applications that maintain a significant
      amount of state (such as an obstacle tracker). These
      applications should reset their internal state when a seek
      occurs: this can be detected by a large discontinuity in the
      timestamps of an physically grounded event stream.</listitem>
  </itemizedlist>
</refsect1>
</refentry>
