Skip to main content
Version: 2.0.0

Basic Concept

NameDescriptionOther
StandardStandard Architecture, contains all InLong components such as InLong Agent, Manager, MQ, Sort, DashboardSupport Data Ingestion, Data Synchronization and Data Subscription at the same time
LightweightLightweight Architecture, contains only one component of InLong Sort, which also can be used with Manager, DashboardThe lightweight architecture is simple and flexible, only support Data Synchronization
Data IngestionData ingestion is the process of moving data from a source into a landing area or an object store where it can be used for ad hoc queries and analytics
Data SynchronizationData synchronization is the process of establishing consistency between source and target data stores, and the continuous harmonization of the data over time
Data SubscriptionData Subscription provides subscribers bulk data feeds of the data they are entitled to access
GroupData Streams Group, it contains multiple data streams, and one Group represents one data business unit.Group has attributes such as ID and Name.
StreamData Stream, a stream has a specific data source, data format and data sink.Stream has attributes such as ID, Name, and data fields.
NodeData Node, including Extract Node and Load Node, stands for the data source and sink types separately.
InLongMsgInLong data format, if you consume message directly from the message queue, you need to perform InLongMsg parsing first.
ClusterEach component can form a single cluster.Contains cluster name, label, necessary information for each component, etc.
TagClusters of different components can use the same tag to represent a set of data stream execution units.Currently tag are only available for clusters
AgentThe standard architecture uses Agent for data collection, and Agent represents different types of collection capabilities.It contains File Agent, SQL Agent, Binlog Agent, etc.
DataProxyForward received data to different message queues.Supports data transmission blocking, placing retransmission.
SortData stream sorting.Sort-flink based on Flink, sort-standalone for local sorting.
TubeMQInLong's self-developed message queuing serviceIt can also be called Tube, with low-cost, high-performance features.
PulsarApache Pulsar, a high-performance, high-consistency message queue service