Standard | Standard Architecture, contains all InLong components such as InLong Agent, Manager, MQ, Sort, Dashboard | Support Data Ingestion , Data Synchronization and Data Subscription at the same time |
Lightweight | Lightweight Architecture, contains only one component of InLong Sort, which also can be used with Manager, Dashboard | The lightweight architecture is simple and flexible, only support Data Synchronization |
Data Ingestion | Data ingestion is the process of moving data from a source into a landing area or an object store where it can be used for ad hoc queries and analytics | |
Data Synchronization | Data synchronization is the process of establishing consistency between source and target data stores, and the continuous harmonization of the data over time | |
Data Subscription | Data Subscription provides subscribers bulk data feeds of the data they are entitled to access | |
Group | Data Streams Group, it contains multiple data streams, and one Group represents one data business unit. | Group has attributes such as ID and Name. |
Stream | Data Stream, a stream has a specific data source, data format and data sink. | Stream has attributes such as ID, Name, and data fields. |
Node | Data Node, including Extract Node and Load Node , stands for the data source and sink types separately. | |
InLongMsg | InLong data format, if you consume message directly from the message queue, you need to perform InLongMsg parsing first. | |
Cluster | Each component can form a single cluster. | Contains cluster name, label, necessary information for each component, etc. |
Tag | Clusters of different components can use the same tag to represent a set of data stream execution units. | Currently tag are only available for clusters |
Agent | The standard architecture uses Agent for data collection, and Agent represents different types of collection capabilities. | It contains File Agent, SQL Agent, Binlog Agent, etc. |
DataProxy | Forward received data to different message queues. | Supports data transmission blocking, placing retransmission. |
Sort | Data stream sorting. | Sort-flink based on Flink, sort-standalone for local sorting. |
TubeMQ | InLong's self-developed message queuing service | It can also be called Tube, with low-cost, high-performance features. |
Pulsar | Apache Pulsar, a high-performance, high-consistency message queue service | |