DataSerializationException caused by java.net.SocketException: Broken pipe (Write failed)

[Shashank11-S]

High Level Area Labels:

• client-v2

Specific Area Labels:

• client-insert
• client-pojo-serde

Description

I have a clickhouse cluster of 7 machines, I am sharding the data manually.
According to the sharding strategy, I distribute the records across que and for each que there is a separate thread which will process the file.
Each Thread do the below things

• poll data of size around 5000
• insert to clickhouse (code is pasted below)

Steps to reproduce

I don't have the steps to reproduce, since it is happening at random time.

Error Log or Exception StackTrace

com.clickhouse.client.api.serde.DataSerializationException: Failed to serialize data 'Record(col1=xxx, col2=7.52, col3=2026-01-27 20:15:00.0, col4=XXXXXXXXX, col5=x.x.x.x, col6=161, col7=TL, col8=xxx, col9=Integer32, col10=x.x.x.x.x, col11=xxxxxx, col12=10, col13=, col14=, col15=, col16=, col17=, col18=, col19=, col20=, col21=0, col22=Y, col23=3, col24=553, col25=NA, col26=2026-01-27 20:15:07.216, col27=2026-01-27 20:15:06.663, col28=2026-01-28 10:46:04.584)' with serializer 'com.clickhouse.client.api.serde.POJOSerDe$$Lambda$501/1285317409@701d586a'
	at com.clickhouse.client.api.Client.lambda$insert$0(Client.java:1270) ~[client-v2-0.9.4.jar:client-v2 0.9.4 (revision: 39bd998)]
	at org.apache.hc.core5.http.io.entity.EntityTemplate.writeTo(EntityTemplate.java:106) ~[httpcore5-5.3.4.jar:5.3.4]
	at org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:285) ~[httpcore5-5.3.4.jar:5.3.4]
	at org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:161) ~[httpcore5-5.3.4.jar:5.3.4]
	at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.lambda$execute$0(InternalExecRuntime.java:236) ~[httpclient5-5.4.4.jar:5.4.4]
	at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$InternalConnectionEndpoint.execute(PoolingHttpClientConnectionManager.java:791) ~[httpclient5-5.4.4.jar:5.4.4]
	at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.execute(InternalExecRuntime.java:233) ~[httpclient5-5.4.4.jar:5.4.4]
	at org.apache.hc.client5.http.impl.classic.MainClientExec.execute(MainClientExec.java:120) ~[httpclient5-5.4.4.jar:5.4.4]
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.4.4.jar:5.4.4]
	at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:198) ~[httpclient5-5.4.4.jar:5.4.4]
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.4.4.jar:5.4.4]
	at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192) ~[httpclient5-5.4.4.jar:5.4.4]
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.4.4.jar:5.4.4]
	at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:113) ~[httpclient5-5.4.4.jar:5.4.4]
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.4.4.jar:5.4.4]
	at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:110) ~[httpclient5-5.4.4.jar:5.4.4]
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.4.4.jar:5.4.4]
	at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:183) ~[httpclient5-5.4.4.jar:5.4.4]
	at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:87) ~[httpclient5-5.4.4.jar:5.4.4]
	at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:55) ~[httpclient5-5.4.4.jar:5.4.4]
	at org.apache.hc.client5.http.classic.HttpClient.executeOpen(HttpClient.java:183) ~[httpclient5-5.4.4.jar:5.4.4]
	at com.clickhouse.client.api.internal.HttpAPIClientHelper.executeRequest(HttpAPIClientHelper.java:453) ~[client-v2-0.9.4.jar:client-v2 0.9.4 (revision: 39bd998)]
	at com.clickhouse.client.api.Client.lambda$insert$1(Client.java:1257) ~[client-v2-0.9.4.jar:client-v2 0.9.4 (revision: 39bd998)]
	at com.clickhouse.client.api.Client.runAsyncOperation(Client.java:2014) ~[client-v2-0.9.4.jar:client-v2 0.9.4 (revision: 39bd998)]
	at com.clickhouse.client.api.Client.insert(Client.java:1307) ~[client-v2-0.9.4.jar:client-v2 0.9.4 (revision: 39bd998)]
	at com.clickhouse.client.api.Client.insert(Client.java:1182) ~[client-v2-0.9.4.jar:client-v2 0.9.4 (revision: 39bd998)]
	at ipm.clickHouse.data.processing.RecordsInserter.insertBatch(RecordsInserter.java:39) ~[clickhouse-ingestion.jar:?]
	at ipm.clickHouse.data.processing.InsertWorker.flushShard(InsertWorker.java:192) ~[clickhouse-ingestion.jar:?]
	at ipm.clickHouse.data.processing.InsertWorker.lambda$startShardProcessor$4(InsertWorker.java:156) ~[clickhouse-ingestion.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_402]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_402]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_402]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_402]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_402]
	Suppressed: java.net.SocketException: Broken pipe (Write failed)
		at java.net.SocketOutputStream.socketWrite0(Native Method) ~[?:1.8.0_402]
		at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) ~[?:1.8.0_402]
		at java.net.SocketOutputStream.write(SocketOutputStream.java:155) ~[?:1.8.0_402]
		at org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection$1.write(DefaultBHttpClientConnection.java:259) ~[httpcore5-5.3.4.jar:5.3.4]
		at org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:117) ~[httpcore5-5.3.4.jar:5.3.4]
		at org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:150) ~[httpcore5-5.3.4.jar:5.3.4]
		at org.apache.hc.core5.http.impl.io.ChunkedOutputStream.flushCache(ChunkedOutputStream.java:127) ~[httpcore5-5.3.4.jar:5.3.4]
		at org.apache.hc.core5.http.impl.io.ChunkedOutputStream.finish(ChunkedOutputStream.java:184) ~[httpcore5-5.3.4.jar:5.3.4]
		at org.apache.hc.core5.http.impl.io.ChunkedOutputStream.close(ChunkedOutputStream.java:245) ~[httpcore5-5.3.4.jar:5.3.4]
		at org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:286) ~[httpcore5-5.3.4.jar:5.3.4]
		at org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:161) ~[httpcore5-5.3.4.jar:5.3.4]
		at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.lambda$execute$0(InternalExecRuntime.java:236) ~[httpclient5-5.4.4.jar:5.4.4]
		at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$InternalConnectionEndpoint.execute(PoolingHttpClientConnectionManager.java:791) ~[httpclient5-5.4.4.jar:5.4.4]
		at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.execute(InternalExecRuntime.java:233) ~[httpclient5-5.4.4.jar:5.4.4]
		at org.apache.hc.client5.http.impl.classic.MainClientExec.execute(MainClientExec.java:120) ~[httpclient5-5.4.4.jar:5.4.4]
		at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.4.4.jar:5.4.4]
		at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:198) ~[httpclient5-5.4.4.jar:5.4.4]
		at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.4.4.jar:5.4.4]
		at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192) ~[httpclient5-5.4.4.jar:5.4.4]
		at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.4.4.jar:5.4.4]
		at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:113) ~[httpclient5-5.4.4.jar:5.4.4]
		at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.4.4.jar:5.4.4]
		at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:110) ~[httpclient5-5.4.4.jar:5.4.4]
		at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.4.4.jar:5.4.4]
		at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:183) ~[httpclient5-5.4.4.jar:5.4.4]
		at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:87) ~[httpclient5-5.4.4.jar:5.4.4]
		at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:55) ~[httpclient5-5.4.4.jar:5.4.4]
		at org.apache.hc.client5.http.classic.HttpClient.executeOpen(HttpClient.java:183) ~[httpclient5-5.4.4.jar:5.4.4]
		at com.clickhouse.client.api.internal.HttpAPIClientHelper.executeRequest(HttpAPIClientHelper.java:453) ~[client-v2-0.9.4.jar:client-v2 0.9.4 (revision: 39bd998)]
		at com.clickhouse.client.api.Client.lambda$insert$1(Client.java:1257) ~[client-v2-0.9.4.jar:client-v2 0.9.4 (revision: 39bd998)]
		at com.clickhouse.client.api.Client.runAsyncOperation(Client.java:2014) ~[client-v2-0.9.4.jar:client-v2 0.9.4 (revision: 39bd998)]
		at com.clickhouse.client.api.Client.insert(Client.java:1307) ~[client-v2-0.9.4.jar:client-v2 0.9.4 (revision: 39bd998)]
		at com.clickhouse.client.api.Client.insert(Client.java:1182) ~[client-v2-0.9.4.jar:client-v2 0.9.4 (revision: 39bd998)]
		at ipm.clickHouse.data.processing.RecordsInserter.insertBatch(RecordsInserter.java:39) ~[clickhouse-ingestion.jar:?]
		at ipm.clickHouse.data.processing.InsertWorker.flushShard(InsertWorker.java:192) ~[clickhouse-ingestion.jar:?]
		at ipm.clickHouse.data.processing.InsertWorker.lambda$startShardProcessor$4(InsertWorker.java:156) ~[clickhouse-ingestion.jar:?]
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_402]
		at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_402]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_402]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_402]
		at java.lang.Thread.run(Thread.java:750) [?:1.8.0_402]
Caused by: java.net.SocketException: Broken pipe (Write failed)
	at java.net.SocketOutputStream.socketWrite0(Native Method) ~[?:1.8.0_402]
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) ~[?:1.8.0_402]
	at java.net.SocketOutputStream.write(SocketOutputStream.java:155) ~[?:1.8.0_402]
	at org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection$1.write(DefaultBHttpClientConnection.java:259) ~[httpcore5-5.3.4.jar:5.3.4]
	at org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:117) ~[httpcore5-5.3.4.jar:5.3.4]
	at org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:150) ~[httpcore5-5.3.4.jar:5.3.4]
	at org.apache.hc.core5.http.impl.io.ChunkedOutputStream.flushCacheWithAppend(ChunkedOutputStream.java:142) ~[httpcore5-5.3.4.jar:5.3.4]
	at org.apache.hc.core5.http.impl.io.ChunkedOutputStream.write(ChunkedOutputStream.java:222) ~[httpcore5-5.3.4.jar:5.3.4]
	at org.apache.hc.core5.http.impl.io.ChunkedOutputStream.write(ChunkedOutputStream.java:209) ~[httpcore5-5.3.4.jar:5.3.4]
	at com.clickhouse.data.format.BinaryStreamUtils.writeString(BinaryStreamUtils.java:1649) ~[clickhouse-data-0.9.4.jar:clickhouse-data 0.9.4 (revision: 39bd998)]
	at com.clickhouse.data.format.BinaryStreamUtils.writeString(BinaryStreamUtils.java:1633) ~[clickhouse-data-0.9.4.jar:clickhouse-data 0.9.4 (revision: 39bd998)]
	at com.clickhouse.client.api.data_formats.internal.SerializerUtils.serializePrimitiveData(SerializerUtils.java:548) ~[client-v2-0.9.4.jar:client-v2 0.9.4 (revision: 39bd998)]
	at com.clickhouse.client.api.data_formats.internal.SerializerUtils.serializeData(SerializerUtils.java:104) ~[client-v2-0.9.4.jar:client-v2 0.9.4 (revision: 39bd998)]
	at com.clickhouse.client.api.serde.POJOSerDe.lambda$registerClass$0(POJOSerDe.java:66) ~[client-v2-0.9.4.jar:client-v2 0.9.4 (revision: 39bd998)]
	at com.clickhouse.client.api.Client.lambda$insert$0(Client.java:1268) ~[client-v2-0.9.4.jar:client-v2 0.9.4 (revision: 39bd998)]
	... 33 more
 

Expected Behaviour

It should run without throwing any Broken pipe exception.

Code Example

Client client = DBClientManager.getInstance().getWriteServer(shardKey);
client.register(type, client.getTableSchema(tableName));
try (InsertResponse response = client.insert(tableName, (List<?>) batch).get()) {
    long timeTakenInMillis = TimeUnit.NANOSECONDS.toMillis(response.getServerTime());
    long noOfRowsWritten = response.getWrittenRows();
    logger.info("Time taken to insert {} records for shard {} is {} ms", noOfRowsWritten, shardKey, timeTakenInMillis);
} catch (ClientException e) {
    logger.error("Client throws Exception while inserting data to shard {} and sample of the batch is {}", shardKey, batch.iterator().next(), e);
    throw e;
} finally {
    client.close();
}

Configuration

Client Configuration

Client.Builder()
    .addEndpoint(url)
    .setUsername(clickHouseConfig.getUserName())
    .setPassword(clickHouseConfig.getPassword())
    .setMaxConnections(clickHouseConfig.getMaxConnections()) //100
    .setSocketTimeout(clickHouseConfig.getSocketTimeout()) // increased to 300000
    .setDefaultDatabase(clickHouseConfig.getDatabaseName())
    .retryOnFailures(ClientFaultCause.values())
    .setMaxRetries(3)
    .setLZ4UncompressedBufferSize(1058576)
    .setSocketRcvbuf(500_000)
    .setSocketTcpNodelay(true)
    .setSocketSndbuf(500_000)
    .setClientNetworkBufferSize(500_000)
    .allowBinaryReaderToReuseBuffers(true)
    .build();

Environment

• Client version: 0.9.4
• Language version: Java 1.8
• OS: Redhat

ClickHouse Server

• ClickHouse Server version: 25.1.8.25
• CREATE TABLE statements for tables involved:

CREATE TABLE db.table_local
(
    `col1` LowCardinality(String),
    `col2` Nullable(String),
    `col3` DateTime('Asia/Kolkata'),
    `col4` String,
    `col5` String,
    `col6` String,
    `col7` LowCardinality(String),
    `col8` LowCardinality(String),
    `col9` Nullable(String),
    `col10` String,
    `col11` String,
    `col12` Nullable(String),
    `col13` Nullable(String),
    `col14` Nullable(String),
    `col15` Nullable(String),
    `col16` Nullable(String),
    `col17` Nullable(String),
    `col18` Nullable(String),
    `col19` Nullable(String),
    `col20` Nullable(String),
    `col21` Nullable(String),
    `col22` Nullable(String),
    `col23` Nullable(String),
    `col24` Nullable(String),
    `col25` Nullable(String),
    `col26` Nullable(DateTime64(3, 'Asia/Kolkata')),
    `col27` Nullable(DateTime64(3, 'Asia/Kolkata')),
    `col28` DateTime('Asia/Kolkata')
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}')
PARTITION BY toDate(timestamp)
PRIMARY KEY (col1, col10, col11, col4)
ORDER BY (col1, col10, col11, col4, col28)
SETTINGS index_granularity = 8192

Please let me know, If any extra info is needed.

[Shashank11-S]

Since the log printed the record, I searched for the specific record in clickhouse, I couldn't find it
Follow up questions

• I cannot find any Retry attempts in log, why ?
• Did I lose whole batch records ?
• I am still not able to figure whether this is a data issue or the network issue, please help on this.
• I tried to insert the record mentioned in the Exception (By converting manually into POJO object, and insert single object to list), It worked. This makes me think this is not an issue with the serialization.