ClickHouse
Overview
The ClickHouse Load Node
supports to write data into ClickHouse database. This document describes how to set up the ClickHouse Load
Node to run SQL queries against ClickHouse database.
Supported Version
Load Node | Driver | Group Id | Artifact Id | JAR |
---|---|---|---|---|
ClickHouse | ClickHouse | ru.yandex.clickhouse | clickhouse-jdbc | Download |
Dependencies
In order to set up the ClickHouse Load Node
, the following provides dependency information for both projects using a
build automation tool (such as Maven or SBT) and SQL Client with Sort Connectors JAR bundles.
Maven dependency
<dependency>
<groupId>org.apache.inlong</groupId>
<artifactId>sort-connector-jdbc</artifactId>
<version>1.13.0-SNAPSHOT</version>
</dependency>
How to create a ClickHouse Load Node
Usage for SQL API
-- MySQL extract node
CREATE TABLE `mysql_extract_table`(
PRIMARY KEY (`id`) NOT ENFORCED,
`id` BIGINT,
`name` STRING,
`age` INT
) WITH (
'connector' = 'mysql-cdc-inlong',
'url' = 'jdbc:mysql://localhost:3306/read',
'username' = 'inlong',
'password' = 'inlong',
'table-name' = 'user'
)
-- ClickHouse load node
CREATE TABLE `clickhouse_load_table`(
PRIMARY KEY (`id`) NOT ENFORCED,
`id` BIGINT,
`name` STRING,
`age` INT
) WITH (
'connector' = 'jdbc-inlong',
'dialect-impl' = 'org.apache.inlong.sort.jdbc.dialect.ClickHouseDialect',
'url' = 'jdbc:clickhouse://localhost:8123/demo',
'username' = 'inlong',
'password' = 'inlong',
'table-name' = 'demo.user'
)
-- write data into ClickHouse
INSERT INTO clickhouse_load_table
SELECT id, name , age FROM mysql_extract_table;
Usage for InLong Dashboard
When creating a data flow, select ClickHouse
for the data stream direction, and click "Add" to configure it.
Usage for InLong Manager Client
TODO: It will be supported in the future.
ClickHouse Load Node Options
Option | Required | Default | Type | Description |
---|---|---|---|---|
connector | required | (none) | String | Specify what connector to use, here should be 'jdbc-inlong'. |
url | required | (none) | String | The JDBC database url. |
dialect-impl | required | (none) | String | org.apache.inlong.sort.jdbc.dialect.ClickHouseDialect |
table-name | required | (none) | String | The name of JDBC table to connect, for example database.tableName |
driver | optional | (none) | String | The class name of the JDBC driver to use to connect to this URL, if not set, it will automatically be derived from the URL. |
username | optional | (none) | String | The JDBC user name. 'username' and 'password' must both be specified if any of them is specified. |
password | optional | (none) | String | The JDBC password. |
connection.max-retry-timeout | optional | 60s | Duration | Maximum timeout between retries. The timeout should be in second granularity and shouldn't be smaller than 1 second. |
sink.buffer-flush.max-rows | optional | 100 | Integer | The max size of buffered records before flush. Can be set to zero to disable it. |
sink.buffer-flush.interval | optional | 1s | Duration | The flush interval mills, over this time, asynchronous threads will flush data. Can be set to '0' to disable it. Note, 'sink.buffer-flush.max-rows' can be set to '0' with the flush interval set allowing for complete async processing of buffered actions. |
sink.max-retries | optional | 3 | Integer | The max retry times if writing records to database failed. |
sink.parallelism | optional | (none) | Integer | Defines the parallelism of the JDBC sink operator. By default, the parallelism is determined by the framework using the same parallelism of the upstream chained operator. |
sink.ignore.changelog | optional | false | Boolean | Ignore all RowKind , ingest them as INSERT . |
inlong.metric.labels | optional | (none) | String | Inlong metric label, format of value is groupId={groupId} &streamId={streamId} &nodeId={nodeId} . |
Data Type Mapping
ClickHouse type | Flink SQL type |
---|---|
String | CHAR |
String IP UUID | VARCHAR |
String EnumL | STRING |
UInt8 | BOOLEAN |
FixedString | BYTES |
Decimal Int128 Int256 UInt64 UInt128 UInt256 | DECIMAL |
Int8 | TINYINT |
Int16 UInt8 | SMALLINT |
Int32 UInt16 Interval | INTEGER |
Int64 UInt32 | BIGINT |
Float32 | FLOAT |
Date | DATE |
DateTime | TIME |
DateTime | TIMESTAMP |
DateTime | TIMESTAMP_LTZ |
Int32 | INTERVAL_YEAR_MONTH |
Int64 | INTERVAL_DAY_TIME |
Array | ARRAY |
Map | MAP |
Not supported | ROW |
Not supported | MULTISET |
Not supported | RAW |