Kafka Compression Performance Tests
End-to-End compression model which means that
Consumer are doing the compression and de-compression jobs. This feature enables the reduction of on-the-fly network costs and the
Broker will increase its cpu load.
|2.5 GHz Intel Core i7||16GB||512GB SSD|
|0.8.2.1||1.7.0u75||2.11||1||1||-Xms4G -Xmx4G -Xmn2G|
The content I used to send to
Kafka is a nginx log which contains 607,781 lines and 200MB. Each line is like below:
127.0.0.1 - - [24/Mar/2015:15:57:09 +0800] "GET /login?gotype=2 HTTP/1.1" "0.002" 200 3177 "http://abc.com/URLhtml" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET4.0C; .NET4.0E)" 127.0.0.1 foo.com foo-hostname
$wc -l passport.access.log
Kafka Producer Configuration
The first test is going to figure out the fact that the value we defined in
batch.size is not the exactly batch size
Producer will use. I got the real batch size of
Producer by using its metrics API. Tests have been done with
batch.size of 200, 500, 1000, 1200, 1500, 2000. Chart below is the result:
We can see clearly that
batch.size is a smaller than
batch-size-avg(it is the excatly batch size
Producer used). And while the
batch.size is 200 and 500, the
batch-size-avg is almost the same - aroud 370. That is to say,
batch-size-avg is not only settled by the parameter
batch.size, furthermore, it is also related to other factor.
All tests have been done in different
batch.size of 500, 1000, 1200, 1500 and 2000. Why I didn’t use
batch.size of 200 because the phenomenon in the above paragraph.
Let’s take a look on the
The throughput rised along with the increasing of
batch.size. And the latency rised as well. Let me make a simple math calculation. I use the
batch.size of 500 for the base number and get the percentage of throughput and latency increasement.
The throughtput of
Producer can get much higher with the increasing of
batch.size. And at the meantime, latency will increas in a very tiny level.
Producer Performance with Compression
Kafka support three different compression type：
Because of the box I use to test, when I use
lz4 in testing, I got OOM exception. I think the casuse of this exception is the Java code will get all lines of the
passport.access.log and send to
Kafka in a very short time. So I used
batch.size of 2000, 2500, 3000, 3500, 4000, 4500 and 5000. Using big
batch.size will increass the latency and this will give JVM some time to do the GC jobs.
Below is the result
The value is better when it is smaller
|Average Compression Rate||100%||19.21%||78.19%||31.37%|
When I enabled the compression, the througput decreased. Let’s see the result.
|Average Latency ms||0.25||4.97||0.41||0.66|
Consumer Performance with Compression
Kafka 0.8.2.1 has not enabled the new
Consumer, so I cannot get the detailed metrics like I have got in
Producer. So in this test, I use the total time
Consumer used to consumes a fixed number of messages to do the benchmark.
The messages are
test.access.log in twice size. Below is the result
|Time Cost ms||3218||5374||5216||4507|
|Time Cost Increase Rate||100%||167%||162.09%||140.06%|
We can see
lz4 is the fastest.
lz4 is the best choice in compression rate and both performance of
Consumer. Below is the comparison of
lz4 performance and base line performance. For simplified the chart, this only shows the result with
batch.size of 5000.