Hub owner: PEDRO DEBEVERE - Last updated on: 16th September 2021
Streaming Telemetry, Flow Monitoring and Network Traffic Flow management are technologies used to find out exactly what's going through your network. Check out this expert hub to get insights on different aspects and how streaming telemetry and flow monitoring is used in different industries.
- Blog: Simplify network operations with telemetry streaming data
- Blog: Proactively monitoring and analyzing media flow in real-time
- Blog: Monitoring high-traffic networks the easy way
- Use Case: Media flow tracking & analytics (TPC Truck APP)
- Use Case: Flow Monitoring - DataMiner Dojo (DataMiner Dashboards)
- Use Case: SFlow Manager (Cisco / Arista / etc.)
- Use Case: NetFlow
- Use Case: Network Topology (Multicast Flow Tracking)
- Use Case: TAP Aggregation, Wireshark and EBU List Tool
- Technology Partner Webinar Video: Arista - Advanced Automation, Orchestration & Visibility in Media Production Networks - DataMiner Dojo (from min 35:00 - TAP aggregation and flow tracking)
- Use Case: Cisco Nexus (Multicast Routing Tables)
- Use Case: Arista Manager Connector (Routing Tables)
- Exhibition Video: NAB 2019 - PTP management and media flow monitoring
- Blog: Multicast flow monitoring and path visualization for ALL-IP Broadcast Infrastructures
- Webinar 2019: All-IP SMPTE-2110 infrastructure and media-over-IP monitoring
- Use Case Video: Integrations - TPC: Overview OB UHD truck-app
OpenConfig is an informal working group with members such as Google, Microsoft, AT&T and British Telecom. An important aspect of the OpenConfig initiative is to create a set of vendor-neutral data models. These data models are designed to take actual operator requirements and needs into account. In addition to data models, OpenConfig also defines communication protocols such as gNMI and gNOI.
OpenConfig - Useful Links
- Blog: Simplify Network Operations With Streaming Telemetry Data
- Official OpenConfig website
- OpenConfig GitHub page
- Example use case DataMiner and OpenConfig
OpenConfig - Data models
OpenConfig data models are defined using YANG, a data modeling language that originates from NETCONF, developed by the NETMOD (IETF) working group.
- RFC 6020 YANG - A Data Modeling Language for the Network Configuration Protocol (NETCONF)
- RFC 7950 The YANG 1.1 Data Modeling Language
YANG is a rich modeling language that supports concepts such as containers, lists, leaves, custom types, etc. You can think of a YANG model as a tree structure where each node or leaf in the tree has a specific path.
Alternatively, tools such as Pyang can be used to visualize or validate YANG models. Below is a visual representation of the OpenConfig-interfaces YANG data model using Pyang.
+--rw interfaces +--rw interface* [name] +--rw name -> ../config/name +--rw config | +--rw name? string | +--rw type identityref | +--rw mtu? uint16 | +--rw loopback-mode? boolean | +--rw description? string | +--rw enabled? boolean +--ro state | +--ro name? string | +--ro type identityref | +--ro mtu? uint16 | +--ro loopback-mode? boolean | +--ro description? string | +--ro enabled? boolean | +--ro ifindex? uint32 | +--ro admin-status enumeration | +--ro oper-status enumeration | +--ro last-change? oc-types:timeticks64 | +--ro logical? boolean | +--ro counters | +--ro in-octets? oc-yang:counter64 | +--ro in-pkts? oc-yang:counter64 | +--ro in-unicast-pkts? oc-yang:counter64 | +--ro in-broadcast-pkts? oc-yang:counter64 | +--ro in-multicast-pkts? oc-yang:counter64 | +--ro in-discards? oc-yang:counter64 | +--ro in-errors? oc-yang:counter64 | +--ro in-unknown-protos? oc-yang:counter64 | +--ro in-fcs-errors? oc-yang:counter64 | +--ro out-octets? oc-yang:counter64 | +--ro out-pkts? oc-yang:counter64 | +--ro out-unicast-pkts? oc-yang:counter64 | +--ro out-broadcast-pkts? oc-yang:counter64 | +--ro out-multicast-pkts? oc-yang:counter64 | +--ro out-discards? oc-yang:counter64 | +--ro out-errors? oc-yang:counter64 | +--ro carrier-transitions? oc-yang:counter64 | +--ro last-clear? oc-types:timeticks64 +--rw hold-time | +--rw config | | +--rw up? uint32 | | +--rw down? uint32 | +--ro state | +--ro up? uint32 | +--ro down? uint32 +--rw subinterfaces +--rw subinterface* [index] +--rw index -> ../config/index +--rw config | +--rw index? uint32 | +--rw description? string | +--rw enabled? boolean +--ro state +--ro index? uint32 +--ro description? string +--ro enabled? boolean +--ro name? string +--ro ifindex? uint32 +--ro admin-status enumeration +--ro oper-status enumeration +--ro last-change? oc-types:timeticks64 +--ro logical? boolean +--ro counters +--ro in-octets? oc-yang:counter64 +--ro in-pkts? oc-yang:counter64 +--ro in-unicast-pkts? oc-yang:counter64 +--ro in-broadcast-pkts? oc-yang:counter64 +--ro in-multicast-pkts? oc-yang:counter64 +--ro in-discards? oc-yang:counter64 +--ro in-errors? oc-yang:counter64 +--ro in-unknown-protos? oc-yang:counter64 +--ro in-fcs-errors? oc-yang:counter64 +--ro out-octets? oc-yang:counter64 +--ro out-pkts? oc-yang:counter64 +--ro out-unicast-pkts? oc-yang:counter64 +--ro out-broadcast-pkts? oc-yang:counter64 +--ro out-multicast-pkts? oc-yang:counter64 +--ro out-discards? oc-yang:counter64 +--ro out-errors? oc-yang:counter64 +--ro carrier-transitions? oc-yang:counter64 +--ro last-clear? oc-types:timeticks64
OpenConfig - Deviations
YANG provides ways to augment, extend and define deviations from other YANG models. As it is possible that an implementor does not support everything defined in an OpenConfig YANG model, implementors can provide additional YANG models describing any deviations.
For example, for the CISCO NX-OS 9.3(5), multiple deviations are defined. The Cisco-nx-OpenConfig interfaces deviations model states that the /oc-if:interfaces/oc-if:interface/oc-if:state/oc-if:counters/oc-if:carrier-transitions node is not supported. Similarly, Arista defines some deviations from the OpenConfig data models. For example, for Arista EOS, the following model indicates that the /oc-if:interfaces/oc-if:interface/oc-if:state/oc-if:counters/oc-if:last-clear is not supported.
OpenConfig - gRPC, gNMI, gNOI, gNxI
In addition to defining vendor-neutral data models, OpenConfig also defines a communication protocol defined as gRPC services.
OpenConfig - gRPC
gRPC, which stands for “gRPC Remote Procedure Call”, is a remote procedure call framework defined by Google. A gRPC service can define unary RPCs, server or client streaming RPCs and bidirectional streaming RPCs.
gRPC services use protocol buffers (which are also created by Google), as the interface description language (IDL). Protocol buffers are a language- and platform-neutral as well as an extensible way of serializing structured data for use in communication protocols or data storage. You'll find more information about protocol buffers here. And this is the official website.
OpenConfig - gNMI
gNMI is a gRPC service defined by OpenConfig and stands for “gRPC Network Management Interface”. The proto file defining the gNMI service can be found on the OpenConfig GitHub page. The full specification is available here .
The gNMI RPC service provides the following four RPCs:
- Capabilities: Allows a client to obtain the capabilities that are supported by the target (service version, supported models and supported encodings)
- Get: Allows to retrieve a snapshot of data from the target
- Set: Modifies the state of data on the target
- Subscribe: Allows a client to subscribe
OpenConfig - Capabilities
This RPC can be used by clients to obtain more information from a target about its capabilities. The capabilities comprise:
- The version of the gNMI protocol (e.g. 0.7.0)
- The supported encodings (e.g. Json, Proto)
- The supported data models: A list of all the supported models. Each entry defines the name of the model, the version of the model and the organization that published the model
Note that the list of supported models is not restricted to OpenConfig models. For example, implementors typically also define their own proprietary YANG models.
The Get RPC can be used by clients to obtain a snapshot of the requested data (the data requested is provided as a number of paths from the data model tree).
Note that a Get RPC should only be used to obtain small amounts of data from the target. To request larger data sets, it is advised to use a subscription.
The Set RPC can be used to modify the state of a target. A set request supports the deletion, replacement and updating of items. All changes that are defined in a set request are considered as one transaction.
The Subscribe RPC allows a client to create subscriptions. gNMI supports three different subscription modes:
- ONCE: Acts as a single request/response channel: The client sends a subscribe request, which gets processed by the target. The target then responds with a series of update messages and finally closes the RPC
- POLL: This is a long-lasting RPC that can be used for on-demand data retrieval. The target sends a subscribe request indicating the paths it wishes to subscribe to. Every time the client wishes to poll these paths, it sends an additional poll message via this long-lasting RPC. Upon receiving such a poll request, the target then generates update messages accordingly
- STREAM: This is a long-lasting RPC, where the target sends updates for the subscribed paths according to the specified mode. Three modes are supported with a STREAM subscription:
- On change: Data updates are sent when the value of the data item changes
- Sampled: Data updates are sent by the target once per sample interval (as specified by the subscription made by the client)
- Target defined: The client specifies that it is up to the target to decide whether an on-change or sampled subscription mechanism is used
gNMI allows dial-in streaming telemetry: The client connects with the target and subscribes to specific paths. The target then continuously sends updates according to the selected subscription method until the RPC is closed by the client.
- Implementors can put limits on the number of RPCs that can exist simultaneously
- Implementors do not always support all types of subscriptions on all paths. For example, CISCO NX-OS 9.3(5) currently does not support on-change stream subscriptions on octet counters (e.g. in-octets, out-octets) defined in the OpenConfig interfaces YANG model
- When using long-lasting RPCs, it is possible that you will need to configure keepalive settings to avoid the RPCs to be closed by the target. You can find more information here
You can also find more information about gNMI here.
gNOI is another gRPC service defined by OpenConfig, which stands for “gRPC Network Operator Interface”. gNMI and gNOI are collectively referred to as gNxI.
You can find more information related to gNOI here .