C语言短字符串压缩
T-BARBARIANS 人气:0前言
上一篇探索了LZ4的压缩和解压性能,以及对LZ4和ZSTD的压缩、解压性能进行了横向对比。文末的最后也给了一个彩蛋:任意长度的字符串都可以被ZSTD、LZ4之类的压缩算压缩得很好吗?
本篇我们就来一探究竟。
一、通用算法的短字符压缩
开门见山,我们使用一段比较短的文本:Narrator: It is raining today. So, Peppa and George cannot play outside.Peppa: Daddy, it's stopped raining.
使用ZSTD与LZ4分别压缩一下上面这段短文本。下面分别是它们的压缩结果。
ZSTD:
LZ4:
对短文本的压缩,zstd的压缩率很低,lz4压缩后的文本长度尽然超过了原有字符串的长度。这是为什么?说实话在这之前我也没想到。
引用两位大佬的名言:
Are you ok?
What's your problem?
二、短字符串压缩
从上面的结果可以得知,任何压缩算法都有它的使用场景,并不是所有长度的字符串都适合被某种算法压缩。一般原因是通用压缩算法维护了被压缩字符串的,用于字符串还原的相关数据结构,而这些数据结构的长度超过了被压缩短字符串的自身长度。
那么问题来了,“我真的有压缩短字符串的需求,我想体验压缩的极致感,怎么办?”。
短字符压缩算法它来了。这里挑选了3种比较优异的短字符压缩算法,分别是smaz,shoco,以及压轴的unisox2。跟前两章一样,还是从压缩率,压缩和解压缩性能的角度,一起看看他们在短字符压缩场景的各自表现吧。
(1)Smaz
1、Smaz的压缩和解压缩
#include <stdio.h> #include <string.h> #include <iostream> #include "smaz.h" using namespace std; int main() { int buf_len; int com_size; int decom_size; char com_buf[4096] = {0}; char decom_buf[4096] = {0}; char str_buf[1024] = "Narrator: It is raining today. So, Peppa and George cannot play outside.Peppa: Daddy, it's stopped raining."; buf_len = strlen(str_buf); com_size = smaz_compress(str_buf, buf_len, com_buf, 4096); cout << "text size:" << buf_len << endl; cout << "compress text size:" << com_size << endl; cout << "compress ratio:" << (float)buf_len / (float)com_size << endl << endl; decom_size = smaz_decompress(com_buf, com_size, decom_buf, 4096); cout << "decompress text size:" << decom_size << endl; if(strncmp(str_buf, decom_buf, buf_len)) { cout << "decompress text is not equal to source text" << endl; } return 0; }
执行结果如下:
通过smaz压缩后的短字符串长度为77,和源字符串相比,减少了30Byte。
2、Smaz的压缩和解压缩性能
#include <stdio.h> #include <string.h> #include <iostream> #include <sys/time.h> #include "smaz.h" using namespace std; int main() { int cnt = 0; int buf_len; int com_size; int decom_size; timeval st, et; char *com_ptr = NULL; char* decom_ptr = NULL; char str_buf[1024] = "Narrator: It is raining today. So, Peppa and George cannot play outside.Peppa: Daddy, it's stopped raining."; buf_len = strlen(str_buf); gettimeofday(&st, NULL); while(1) { com_ptr = (char *)malloc(buf_len); com_size = smaz_compress(str_buf, buf_len, com_ptr, buf_len); free(com_ptr); cnt++; gettimeofday(&et, NULL); if(et.tv_sec - st.tv_sec >= 10) { break; } } cout << endl <<"compress per second:" << cnt/10 << " times" << endl; cnt = 0; com_ptr = (char *)malloc(buf_len); com_size = smaz_compress(str_buf, buf_len, com_ptr, buf_len); gettimeofday(&st, NULL); while(1) { // decompress length not more than origin buf length decom_ptr = (char *)malloc(buf_len + 1); decom_size = smaz_decompress(com_ptr, com_size, decom_ptr, buf_len + 1); // check decompress length if(buf_len != decom_size) { cout << "decom error" << endl; } free(decom_ptr); cnt++; gettimeofday(&et, NULL); if(et.tv_sec - st.tv_sec >= 10) { break; } } cout << "decompress per second:" << cnt/10 << " times" << endl << endl; free(com_ptr); return 0; }
结果如何?
压缩性能在40w条/S,解压在百万级,好像还不错哈!
(2)Shoco
1、Shoco的压缩和解压缩
#include <stdio.h> #include <string.h> #include <iostream> #include "shoco.h" using namespace std; int main() { int buf_len; int com_size; int decom_size; char com_buf[4096] = {0}; char decom_buf[4096] = {0}; char str_buf[1024] = "Narrator: It is raining today. So, Peppa and George cannot play outside.Peppa: Daddy, it's stopped raining."; buf_len = strlen(str_buf); com_size = shoco_compress(str_buf, buf_len, com_buf, 4096); cout << "text size:" << buf_len << endl; cout << "compress text size:" << com_size << endl; cout << "compress ratio:" << (float)buf_len / (float)com_size << endl << endl; decom_size = shoco_decompress(com_buf, com_size, decom_buf, 4096); cout << "decompress text size:" << decom_size << endl; if(strncmp(str_buf, decom_buf, buf_len)) { cout << "decompress text is not equal to source text" << endl; } return 0; }
执行结果如下:
通过shoco压缩后的短字符串长度为86,和源字符串相比,减少了21Byte。压缩率比smaz要低。
2、Shoco的压缩和解压缩性能
#include <stdio.h> #include <string.h> #include <iostream> #include <sys/time.h> #include "shoco.h" using namespace std; int main() { int cnt = 0; int buf_len; int com_size; int decom_size; timeval st, et; char *com_ptr = NULL; char* decom_ptr = NULL; char str_buf[1024] = "Narrator: It is raining today. So, Peppa and George cannot play outside.Peppa: Daddy, it's stopped raining."; buf_len = strlen(str_buf); gettimeofday(&st, NULL); while(1) { com_ptr = (char *)malloc(buf_len); com_size = shoco_compress(str_buf, buf_len, com_ptr, buf_len); free(com_ptr); cnt++; gettimeofday(&et, NULL); if(et.tv_sec - st.tv_sec >= 10) { break; } } cout << endl <<"compress per second:" << cnt/10 << " times" << endl; cnt = 0; com_ptr = (char *)malloc(buf_len); com_size = shoco_compress(str_buf, buf_len, com_ptr, buf_len); gettimeofday(&st, NULL); while(1) { // decompress length not more than origin buf length decom_ptr = (char *)malloc(buf_len + 1); decom_size = shoco_decompress(com_ptr, com_size, decom_ptr, buf_len + 1); // check decompress length if(buf_len != decom_size) { cout << "decom error" << endl; } free(decom_ptr); cnt++; gettimeofday(&et, NULL); if(et.tv_sec - st.tv_sec >= 10) { break; } } cout << "decompress per second:" << cnt/10 << " times" << endl << endl; free(com_ptr); return 0; }
执行结果如何呢?
holy shit!压缩和解压缩居然都达到了惊人的百万级。就像算法作者们自己说的一样:“在长字符串压缩领域,shoco不想与通用压缩算法竞争,我们的优势是短字符的快速压缩,虽然压缩率很烂!”。这样说,好像也没毛病。
(3)Unisox2
我们再来看看unisox2呢。
1、Unisox2的压缩和解压缩
#include <stdio.h> #include <string.h> #include "unishox2.h" int main() { int buf_len; int com_size; int decom_size; char com_buf[4096] = {0}; char decom_buf[4096] = {0}; char str_buf[1024] = "Narrator: It is raining today. So, Peppa and George cannot play outside.Peppa: Daddy, it's stopped raining."; buf_len = strlen(str_buf); com_size = unishox2_compress_simple(str_buf, buf_len, com_buf); printf("text size:%d\n", buf_len); printf("compress text size:%d\n", com_size); printf("compress ratio:%f\n\n", (float)buf_len / (float)com_size); decom_size = unishox2_decompress_simple(com_buf, com_size, decom_buf); printf("decompress text size:%d\n", decom_size); if(strncmp(str_buf, decom_buf, buf_len)) { printf("decompress text is not equal to source text\n"); } return 0; }
结果如下:
通过Unisox2压缩后的短字符串长度为67,和源字符串相比,减少了40Byte,相当于是打了6折啊!不错不错。
2、Unisox2的压缩和解压缩性能
Unisox2的压缩能力目前来看是三者中最好的,如果他的压缩和解压性能也不错的话,那就真的就比较完美了。再一起看看Unisox2的压缩和解压性能吧!
#include <stdio.h> #include <string.h> #include <malloc.h> #include <sys/time.h> #include "unishox2.h" int main() { int cnt = 0; int buf_len; int com_size; int decom_size; struct timeval st, et; char *com_ptr = NULL; char* decom_ptr = NULL; char str_buf[1024] = "Narrator: It is raining today. So, Peppa and George cannot play outside.Peppa: Daddy, it's stopped raining."; buf_len = strlen(str_buf); gettimeofday(&st, NULL); while(1) { com_ptr = (char *)malloc(buf_len); com_size = unishox2_compress_simple(str_buf, buf_len, com_ptr); free(com_ptr); cnt++; gettimeofday(&et, NULL); if(et.tv_sec - st.tv_sec >= 10) { break; } } printf("\ncompress per second:%d times\n", cnt/10); cnt = 0; com_ptr = (char *)malloc(buf_len); com_size = unishox2_compress_simple(str_buf, buf_len, com_ptr); gettimeofday(&st, NULL); while(1) { // decompress length not more than origin buf length decom_ptr = (char *)malloc(buf_len + 1); decom_size = unishox2_decompress_simple(com_ptr, com_size, decom_ptr); // check decompress length if(buf_len != decom_size) { printf("decom error\n"); } free(decom_ptr); cnt++; gettimeofday(&et, NULL); if(et.tv_sec - st.tv_sec >= 10) { break; } } printf("decompress per second:%d times\n\n", cnt/10); free(com_ptr); return 0; }
执行结果如下:
事与愿违,Unisox2虽然有三个算法中最好的压缩率,可是却也拥有最差的压缩和解压性能。跟前两章分析的不谋而合:有高压缩率,就会损失自身的压缩性能,两者不可兼得。
三、总结
本篇分享了smaz,shoco,unisox2三种短字符串压缩算法,分别探索了它们各自的压缩率与压缩和解压缩性能,结果如下表所示。
表1
shoco的压缩率最低,但是拥有最高的压缩和解压速率;smaz居中;unisox2拥有最高的压缩率,可是它的压缩和解压性能最低。
结论与前两章有关长字符串压缩的分析不谋而合:拥有高压缩率,就会损失自身的压缩性能,两者不可兼得。
实际使用还是看自身需求和环境吧。如果适当压缩就好,那就可以选用shoco,毕竟性能高;想要节约更多的空间,那就选择smaz或者unisox2。
加载全部内容