tmtt/chez-openbsd
Archived
1
0
Fork 0
Code Issues Pull requests Releases 1 Activity

feat: 9.5.9

Browse source
This commit is contained in:
tmtt 2022-07-29 15:12:07 +02:00
parent cb1753732b
commit 35f43a7909
1084 changed files with 558985 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

10
lz4/examples/.gitignore vendored Normal file
View file

@ -0,0 +1,10 @@
/Makefile.lz4*
/printVersion
/doubleBuffer
/dictionaryRandomAccess
/ringBuffer
/ringBufferHC
/lineCompress
/frameCompress
/simpleBuffer
/*.exe

339
lz4/examples/COPYING Normal file
View file

@ -0,0 +1,339 @@
GNU GENERAL PUBLIC LICENSE
Version 2, June 1991
Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The licenses for most software are designed to take away your
freedom to share and change it. By contrast, the GNU General Public
License is intended to guarantee your freedom to share and change free
software--to make sure the software is free for all its users. This
General Public License applies to most of the Free Software
Foundation's software and to any other program whose authors commit to
using it. (Some other Free Software Foundation software is covered by
the GNU Lesser General Public License instead.) You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
this service if you wish), that you receive source code or can get it
if you want it, that you can change the software or use pieces of it
in new free programs; and that you know you can do these things.
To protect your rights, we need to make restrictions that forbid
anyone to deny you these rights or to ask you to surrender the rights.
These restrictions translate to certain responsibilities for you if you
distribute copies of the software, or if you modify it.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must give the recipients all the rights that
you have. You must make sure that they, too, receive or can get the
source code. And you must show them these terms so they know their
rights.
We protect your rights with two steps: (1) copyright the software, and
(2) offer you this license which gives you legal permission to copy,
distribute and/or modify the software.
Also, for each author's protection and ours, we want to make certain
that everyone understands that there is no warranty for this free
software. If the software is modified by someone else and passed on, we
want its recipients to know that what they have is not the original, so
that any problems introduced by others will not reflect on the original
authors' reputations.
Finally, any free program is threatened constantly by software
patents. We wish to avoid the danger that redistributors of a free
program will individually obtain patent licenses, in effect making the
program proprietary. To prevent this, we have made it clear that any
patent must be licensed for everyone's free use or not licensed at all.
The precise terms and conditions for copying, distribution and
modification follow.
GNU GENERAL PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. This License applies to any program or other work which contains
a notice placed by the copyright holder saying it may be distributed
under the terms of this General Public License. The "Program", below,
refers to any such program or work, and a "work based on the Program"
means either the Program or any derivative work under copyright law:
that is to say, a work containing the Program or a portion of it,
either verbatim or with modifications and/or translated into another
language. (Hereinafter, translation is included without limitation in
the term "modification".) Each licensee is addressed as "you".
Activities other than copying, distribution and modification are not
covered by this License; they are outside its scope. The act of
running the Program is not restricted, and the output from the Program
is covered only if its contents constitute a work based on the
Program (independent of having been made by running the Program).
Whether that is true depends on what the Program does.
1. You may copy and distribute verbatim copies of the Program's
source code as you receive it, in any medium, provided that you
conspicuously and appropriately publish on each copy an appropriate
copyright notice and disclaimer of warranty; keep intact all the
notices that refer to this License and to the absence of any warranty;
and give any other recipients of the Program a copy of this License
along with the Program.
You may charge a fee for the physical act of transferring a copy, and
you may at your option offer warranty protection in exchange for a fee.
2. You may modify your copy or copies of the Program or any portion
of it, thus forming a work based on the Program, and copy and
distribute such modifications or work under the terms of Section 1
above, provided that you also meet all of these conditions:
a) You must cause the modified files to carry prominent notices
stating that you changed the files and the date of any change.
b) You must cause any work that you distribute or publish, that in
whole or in part contains or is derived from the Program or any
part thereof, to be licensed as a whole at no charge to all third
parties under the terms of this License.
c) If the modified program normally reads commands interactively
when run, you must cause it, when started running for such
interactive use in the most ordinary way, to print or display an
announcement including an appropriate copyright notice and a
notice that there is no warranty (or else, saying that you provide
a warranty) and that users may redistribute the program under
these conditions, and telling the user how to view a copy of this
License. (Exception: if the Program itself is interactive but
does not normally print such an announcement, your work based on
the Program is not required to print an announcement.)
These requirements apply to the modified work as a whole. If
identifiable sections of that work are not derived from the Program,
and can be reasonably considered independent and separate works in
themselves, then this License, and its terms, do not apply to those
sections when you distribute them as separate works. But when you
distribute the same sections as part of a whole which is a work based
on the Program, the distribution of the whole must be on the terms of
this License, whose permissions for other licensees extend to the
entire whole, and thus to each and every part regardless of who wrote it.
Thus, it is not the intent of this section to claim rights or contest
your rights to work written entirely by you; rather, the intent is to
exercise the right to control the distribution of derivative or
collective works based on the Program.
In addition, mere aggregation of another work not based on the Program
with the Program (or with a work based on the Program) on a volume of
a storage or distribution medium does not bring the other work under
the scope of this License.
3. You may copy and distribute the Program (or a work based on it,
under Section 2) in object code or executable form under the terms of
Sections 1 and 2 above provided that you also do one of the following:
a) Accompany it with the complete corresponding machine-readable
source code, which must be distributed under the terms of Sections
1 and 2 above on a medium customarily used for software interchange; or,
b) Accompany it with a written offer, valid for at least three
years, to give any third party, for a charge no more than your
cost of physically performing source distribution, a complete
machine-readable copy of the corresponding source code, to be
distributed under the terms of Sections 1 and 2 above on a medium
customarily used for software interchange; or,
c) Accompany it with the information you received as to the offer
to distribute corresponding source code. (This alternative is
allowed only for noncommercial distribution and only if you
received the program in object code or executable form with such
an offer, in accord with Subsection b above.)
The source code for a work means the preferred form of the work for
making modifications to it. For an executable work, complete source
code means all the source code for all modules it contains, plus any
associated interface definition files, plus the scripts used to
control compilation and installation of the executable. However, as a
special exception, the source code distributed need not include
anything that is normally distributed (in either source or binary
form) with the major components (compiler, kernel, and so on) of the
operating system on which the executable runs, unless that component
itself accompanies the executable.
If distribution of executable or object code is made by offering
access to copy from a designated place, then offering equivalent
access to copy the source code from the same place counts as
distribution of the source code, even though third parties are not
compelled to copy the source along with the object code.
4. You may not copy, modify, sublicense, or distribute the Program
except as expressly provided under this License. Any attempt
otherwise to copy, modify, sublicense or distribute the Program is
void, and will automatically terminate your rights under this License.
However, parties who have received copies, or rights, from you under
this License will not have their licenses terminated so long as such
parties remain in full compliance.
5. You are not required to accept this License, since you have not
signed it. However, nothing else grants you permission to modify or
distribute the Program or its derivative works. These actions are
prohibited by law if you do not accept this License. Therefore, by
modifying or distributing the Program (or any work based on the
Program), you indicate your acceptance of this License to do so, and
all its terms and conditions for copying, distributing or modifying
the Program or works based on it.
6. Each time you redistribute the Program (or any work based on the
Program), the recipient automatically receives a license from the
original licensor to copy, distribute or modify the Program subject to
these terms and conditions. You may not impose any further
restrictions on the recipients' exercise of the rights granted herein.
You are not responsible for enforcing compliance by third parties to
this License.
7. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot
distribute so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you
may not distribute the Program at all. For example, if a patent
license would not permit royalty-free redistribution of the Program by
all those who receive copies directly or indirectly through you, then
the only way you could satisfy both it and this License would be to
refrain entirely from distribution of the Program.
If any portion of this section is held invalid or unenforceable under
any particular circumstance, the balance of the section is intended to
apply and the section as a whole is intended to apply in other
circumstances.
It is not the purpose of this section to induce you to infringe any
patents or other property right claims or to contest validity of any
such claims; this section has the sole purpose of protecting the
integrity of the free software distribution system, which is
implemented by public license practices. Many people have made
generous contributions to the wide range of software distributed
through that system in reliance on consistent application of that
system; it is up to the author/donor to decide if he or she is willing
to distribute software through any other system and a licensee cannot
impose that choice.
This section is intended to make thoroughly clear what is believed to
be a consequence of the rest of this License.
8. If the distribution and/or use of the Program is restricted in
certain countries either by patents or by copyrighted interfaces, the
original copyright holder who places the Program under this License
may add an explicit geographical distribution limitation excluding
those countries, so that distribution is permitted only in or among
countries not thus excluded. In such case, this License incorporates
the limitation as if written in the body of this License.
9. The Free Software Foundation may publish revised and/or new versions
of the General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the Program
specifies a version number of this License which applies to it and "any
later version", you have the option of following the terms and conditions
either of that version or of any later version published by the Free
Software Foundation. If the Program does not specify a version number of
this License, you may choose any version ever published by the Free Software
Foundation.
10. If you wish to incorporate parts of the Program into other free
programs whose distribution conditions are different, write to the author
to ask for permission. For software which is copyrighted by the Free
Software Foundation, write to the Free Software Foundation; we sometimes
make exceptions for this. Our decision will be guided by the two goals
of preserving the free status of all derivatives of our free software and
of promoting the sharing and reuse of software generally.
NO WARRANTY
11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
REPAIR OR CORRECTION.
12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
convey the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
Also add information on how to contact you by electronic and paper mail.
If the program is interactive, make it output a short notice like this
when it starts in an interactive mode:
Gnomovision version 69, Copyright (C) year name of author
Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, the commands you use may
be called something other than `show w' and `show c'; they could even be
mouse-clicks or menu items--whatever suits your program.
You should also get your employer (if you work as a programmer) or your
school, if any, to sign a "copyright disclaimer" for the program, if
necessary. Here is a sample; alter the names:
Yoyodyne, Inc., hereby disclaims all copyright interest in the program
`Gnomovision' (which makes passes at compilers) written by James Hacker.
<signature of Ty Coon>, 1 April 1989
Ty Coon, President of Vice
This General Public License does not permit incorporating your program into
proprietary programs. If your program is a subroutine library, you may
consider it more useful to permit linking proprietary applications with the
library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License.

232
lz4/examples/HCStreaming_ringBuffer.c Normal file
View file

@ -0,0 +1,232 @@
// LZ4 HC streaming API example : ring buffer
// Based on a previous example by Takayuki Matsuoka
/**************************************
* Compiler Options
**************************************/
#if defined(_MSC_VER) && (_MSC_VER <= 1800) /* Visual Studio <= 2013 */
# define _CRT_SECURE_NO_WARNINGS
# define snprintf sprintf_s
#endif
#define GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__)
#ifdef __GNUC__
# pragma GCC diagnostic ignored "-Wmissing-braces" /* GCC bug 53119 : doesn't accept { 0 } as initializer (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119) */
#endif
/**************************************
* Includes
**************************************/
#include "lz4hc.h"
#include "lz4.h"
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include <assert.h>
enum {
MESSAGE_MAX_BYTES = 1024,
RING_BUFFER_BYTES = 1024 * 8 + MESSAGE_MAX_BYTES,
DEC_BUFFER_BYTES = RING_BUFFER_BYTES + MESSAGE_MAX_BYTES // Intentionally larger to test unsynchronized ring buffers
};
size_t write_int32(FILE* fp, int32_t i) {
return fwrite(&i, sizeof(i), 1, fp);
}
size_t write_bin(FILE* fp, const void* array, int arrayBytes) {
assert(arrayBytes >= 0);
return fwrite(array, 1, (size_t)arrayBytes, fp);
}
size_t read_int32(FILE* fp, int32_t* i) {
return fread(i, sizeof(*i), 1, fp);
}
size_t read_bin(FILE* fp, void* array, int arrayBytes) {
assert(arrayBytes >= 0);
return fread(array, 1, (size_t)arrayBytes, fp);
}
void test_compress(FILE* outFp, FILE* inpFp)
{
LZ4_streamHC_t lz4Stream_body = { 0 };
LZ4_streamHC_t* lz4Stream = &lz4Stream_body;
static char inpBuf[RING_BUFFER_BYTES];
int inpOffset = 0;
for(;;) {
// Read random length ([1,MESSAGE_MAX_BYTES]) data to the ring buffer.
char* const inpPtr = &inpBuf[inpOffset];
const int randomLength = (rand() % MESSAGE_MAX_BYTES) + 1;
const int inpBytes = (int) read_bin(inpFp, inpPtr, randomLength);
if (0 == inpBytes) break;
#define CMPBUFSIZE (LZ4_COMPRESSBOUND(MESSAGE_MAX_BYTES))
{ char cmpBuf[CMPBUFSIZE];
const int cmpBytes = LZ4_compress_HC_continue(lz4Stream, inpPtr, cmpBuf, inpBytes, CMPBUFSIZE);
if(cmpBytes <= 0) break;
write_int32(outFp, cmpBytes);
write_bin(outFp, cmpBuf, cmpBytes);
inpOffset += inpBytes;
// Wraparound the ringbuffer offset
if(inpOffset >= RING_BUFFER_BYTES - MESSAGE_MAX_BYTES)
inpOffset = 0;
}
}
write_int32(outFp, 0);
}
void test_decompress(FILE* outFp, FILE* inpFp)
{
static char decBuf[DEC_BUFFER_BYTES];
int decOffset = 0;
LZ4_streamDecode_t lz4StreamDecode_body = { 0 };
LZ4_streamDecode_t* lz4StreamDecode = &lz4StreamDecode_body;
for(;;) {
int cmpBytes = 0;
char cmpBuf[CMPBUFSIZE];
{ const size_t r0 = read_int32(inpFp, &cmpBytes);
size_t r1;
if(r0 != 1 || cmpBytes <= 0)
break;
r1 = read_bin(inpFp, cmpBuf, cmpBytes);
if(r1 != (size_t) cmpBytes)
break;
}
{ char* const decPtr = &decBuf[decOffset];
const int decBytes = LZ4_decompress_safe_continue(
lz4StreamDecode, cmpBuf, decPtr, cmpBytes, MESSAGE_MAX_BYTES);
if(decBytes <= 0)
break;
decOffset += decBytes;
write_bin(outFp, decPtr, decBytes);
// Wraparound the ringbuffer offset
if(decOffset >= DEC_BUFFER_BYTES - MESSAGE_MAX_BYTES)
decOffset = 0;
}
}
}
// Compare 2 files content
// return 0 if identical
// return ByteNb>0 if different
size_t compare(FILE* f0, FILE* f1)
{
size_t result = 1;
for (;;) {
char b0[65536];
char b1[65536];
const size_t r0 = fread(b0, 1, sizeof(b0), f0);
const size_t r1 = fread(b1, 1, sizeof(b1), f1);
if ((r0==0) && (r1==0)) return 0; // success
if (r0 != r1) {
size_t smallest = r0;
if (r1<r0) smallest = r1;
result += smallest;
break;
}
if (memcmp(b0, b1, r0)) {
unsigned errorPos = 0;
while ((errorPos < r0) && (b0[errorPos]==b1[errorPos])) errorPos++;
result += errorPos;
break;
}
result += sizeof(b0);
}
return result;
}
int main(int argc, const char** argv)
{
char inpFilename[256] = { 0 };
char lz4Filename[256] = { 0 };
char decFilename[256] = { 0 };
unsigned fileID = 1;
unsigned pause = 0;
if(argc < 2) {
printf("Please specify input filename\n");
return 0;
}
if (!strcmp(argv[1], "-p")) { pause = 1; fileID = 2; }
snprintf(inpFilename, 256, "%s", argv[fileID]);
snprintf(lz4Filename, 256, "%s.lz4s-%d", argv[fileID], 9);
snprintf(decFilename, 256, "%s.lz4s-%d.dec", argv[fileID], 9);
printf("input = [%s]\n", inpFilename);
printf("lz4 = [%s]\n", lz4Filename);
printf("decoded = [%s]\n", decFilename);
// compress
{ FILE* const inpFp = fopen(inpFilename, "rb");
FILE* const outFp = fopen(lz4Filename, "wb");
test_compress(outFp, inpFp);
fclose(outFp);
fclose(inpFp);
}
// decompress
{ FILE* const inpFp = fopen(lz4Filename, "rb");
FILE* const outFp = fopen(decFilename, "wb");
test_decompress(outFp, inpFp);
fclose(outFp);
fclose(inpFp);
}
// verify
{ FILE* const inpFp = fopen(inpFilename, "rb");
FILE* const decFp = fopen(decFilename, "rb");
const size_t cmp = compare(inpFp, decFp);
if(0 == cmp) {
printf("Verify : OK\n");
} else {
printf("Verify : NG : error at pos %u\n", (unsigned)cmp-1);
}
fclose(decFp);
fclose(inpFp);
}
if (pause) {
int unused;
printf("Press enter to continue ...\n");
unused = getchar(); (void)unused; /* silence static analyzer */
}
return 0;
}

103
lz4/examples/Makefile Normal file
View file

@ -0,0 +1,103 @@
# ##########################################################################
# LZ4 examples - Makefile
# Copyright (C) Yann Collet 2011-2014
#
# GPL v2 License
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
#
# You can contact the author at :
# - LZ4 source repository : https://github.com/lz4/lz4
# - LZ4 forum froup : https://groups.google.com/forum/#!forum/lz4c
# ##########################################################################
# This makefile compile and test
# example programs, using (mostly) LZ4 streaming library,
# kindly provided by Takayuki Matsuoka
# ##########################################################################
CPPFLAGS += -I../lib
CFLAGS ?= -O3
CFLAGS += -std=gnu99 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Wstrict-prototypes
FLAGS := $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) $(MOREFLAGS)
TESTFILE = Makefile
LZ4DIR := ../lib
LZ4 = ../programs/lz4
include ../Makefile.inc
default: all
all: printVersion doubleBuffer dictionaryRandomAccess ringBuffer ringBufferHC \
lineCompress frameCompress simpleBuffer
$(LZ4DIR)/liblz4.a: $(LZ4DIR)/lz4.c $(LZ4DIR)/lz4hc.c $(LZ4DIR)/lz4frame.c $(LZ4DIR)/lz4.h $(LZ4DIR)/lz4hc.h $(LZ4DIR)/lz4frame.h $(LZ4DIR)/lz4frame_static.h
$(MAKE) -C $(LZ4DIR) liblz4.a
printVersion: printVersion.c $(LZ4DIR)/liblz4.a
$(CC) $(FLAGS) $^ -o $@$(EXT)
doubleBuffer: blockStreaming_doubleBuffer.c $(LZ4DIR)/liblz4.a
$(CC) $(FLAGS) $^ -o $@$(EXT)
dictionaryRandomAccess: dictionaryRandomAccess.c $(LZ4DIR)/liblz4.a
$(CC) $(FLAGS) $^ -o $@$(EXT)
ringBuffer : blockStreaming_ringBuffer.c $(LZ4DIR)/liblz4.a
$(CC) $(FLAGS) $^ -o $@$(EXT)
ringBufferHC: HCStreaming_ringBuffer.c $(LZ4DIR)/liblz4.a
$(CC) $(FLAGS) $^ -o $@$(EXT)
lineCompress: blockStreaming_lineByLine.c $(LZ4DIR)/liblz4.a
$(CC) $(FLAGS) $^ -o $@$(EXT)
frameCompress: frameCompress.c $(LZ4DIR)/liblz4.a
$(CC) $(FLAGS) $^ -o $@$(EXT)
compressFunctions: compress_functions.c $(LZ4DIR)/liblz4.a
$(CC) $(FLAGS) $^ -o $@$(EXT) -lrt
simpleBuffer: simple_buffer.c $(LZ4DIR)/liblz4.a
$(CC) $(FLAGS) $^ -o $@$(EXT)
$(LZ4) :
$(MAKE) -C ../programs lz4
test : all $(LZ4)
@echo "\n=== Print Version ==="
./printVersion$(EXT)
@echo "\n=== Simple compression example ==="
./simpleBuffer$(EXT)
@echo "\n=== Double-buffer ==="
./doubleBuffer$(EXT) $(TESTFILE)
@echo "\n=== Ring Buffer ==="
./ringBuffer$(EXT) $(TESTFILE)
@echo "\n=== Ring Buffer + LZ4 HC ==="
./ringBufferHC$(EXT) $(TESTFILE)
@echo "\n=== Compress line by line ==="
./lineCompress$(EXT) $(TESTFILE)
@echo "\n=== Dictionary Random Access ==="
./dictionaryRandomAccess$(EXT) $(TESTFILE) $(TESTFILE) 1100 1400
@echo "\n=== Frame compression ==="
./frameCompress$(EXT) $(TESTFILE)
$(LZ4) -vt $(TESTFILE).lz4
clean:
@rm -f core *.o *.dec *-0 *-9 *-8192 *.lz4s *.lz4 \
printVersion$(EXT) doubleBuffer$(EXT) dictionaryRandomAccess$(EXT) \
ringBuffer$(EXT) ringBufferHC$(EXT) lineCompress$(EXT) frameCompress$(EXT) \
compressFunctions$(EXT) simpleBuffer$(EXT)
@echo Cleaning completed

11
lz4/examples/README.md Normal file
View file

@ -0,0 +1,11 @@
# LZ4 examples
All examples are GPL-v2 licensed.
## Documents
- [Streaming API Basics](streaming_api_basics.md)
- Examples
- [Double Buffer](blockStreaming_doubleBuffer.md)
- [Line by Line Text Compression](blockStreaming_lineByLine.md)
- [Dictionary Random Access](dictionaryRandomAccess.md)

202
lz4/examples/blockStreaming_doubleBuffer.c Normal file
View file

@ -0,0 +1,202 @@
// LZ4 streaming API example : double buffer
// Copyright : Takayuki Matsuoka
#if defined(_MSC_VER) && (_MSC_VER <= 1800) /* Visual Studio <= 2013 */
# define _CRT_SECURE_NO_WARNINGS
# define snprintf sprintf_s
#endif
#include "lz4.h"
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
enum {
BLOCK_BYTES = 1024 * 8,
// BLOCK_BYTES = 1024 * 64,
};
size_t write_int(FILE* fp, int i) {
return fwrite(&i, sizeof(i), 1, fp);
}
size_t write_bin(FILE* fp, const void* array, size_t arrayBytes) {
return fwrite(array, 1, arrayBytes, fp);
}
size_t read_int(FILE* fp, int* i) {
return fread(i, sizeof(*i), 1, fp);
}
size_t read_bin(FILE* fp, void* array, size_t arrayBytes) {
return fread(array, 1, arrayBytes, fp);
}
void test_compress(FILE* outFp, FILE* inpFp)
{
LZ4_stream_t lz4Stream_body;
LZ4_stream_t* lz4Stream = &lz4Stream_body;
char inpBuf[2][BLOCK_BYTES];
int inpBufIndex = 0;
LZ4_initStream(lz4Stream, sizeof (*lz4Stream));
for(;;) {
char* const inpPtr = inpBuf[inpBufIndex];
const int inpBytes = (int) read_bin(inpFp, inpPtr, BLOCK_BYTES);
if(0 == inpBytes) {
break;
}
{
char cmpBuf[LZ4_COMPRESSBOUND(BLOCK_BYTES)];
const int cmpBytes = LZ4_compress_fast_continue(
lz4Stream, inpPtr, cmpBuf, inpBytes, sizeof(cmpBuf), 1);
if(cmpBytes <= 0) {
break;
}
write_int(outFp, cmpBytes);
write_bin(outFp, cmpBuf, (size_t) cmpBytes);
}
inpBufIndex = (inpBufIndex + 1) % 2;
}
write_int(outFp, 0);
}
void test_decompress(FILE* outFp, FILE* inpFp)
{
LZ4_streamDecode_t lz4StreamDecode_body;
LZ4_streamDecode_t* lz4StreamDecode = &lz4StreamDecode_body;
char decBuf[2][BLOCK_BYTES];
int decBufIndex = 0;
LZ4_setStreamDecode(lz4StreamDecode, NULL, 0);
for(;;) {
char cmpBuf[LZ4_COMPRESSBOUND(BLOCK_BYTES)];
int cmpBytes = 0;
{
const size_t readCount0 = read_int(inpFp, &cmpBytes);
if(readCount0 != 1 || cmpBytes <= 0) {
break;
}
const size_t readCount1 = read_bin(inpFp, cmpBuf, (size_t) cmpBytes);
if(readCount1 != (size_t) cmpBytes) {
break;
}
}
{
char* const decPtr = decBuf[decBufIndex];
const int decBytes = LZ4_decompress_safe_continue(
lz4StreamDecode, cmpBuf, decPtr, cmpBytes, BLOCK_BYTES);
if(decBytes <= 0) {
break;
}
write_bin(outFp, decPtr, (size_t) decBytes);
}
decBufIndex = (decBufIndex + 1) % 2;
}
}
int compare(FILE* fp0, FILE* fp1)
{
int result = 0;
while(0 == result) {
char b0[65536];
char b1[65536];
const size_t r0 = read_bin(fp0, b0, sizeof(b0));
const size_t r1 = read_bin(fp1, b1, sizeof(b1));
result = (int) r0 - (int) r1;
if(0 == r0 || 0 == r1) {
break;
}
if(0 == result) {
result = memcmp(b0, b1, r0);
}
}
return result;
}
int main(int argc, char* argv[])
{
char inpFilename[256] = { 0 };
char lz4Filename[256] = { 0 };
char decFilename[256] = { 0 };
if(argc < 2) {
printf("Please specify input filename\n");
return 0;
}
snprintf(inpFilename, 256, "%s", argv[1]);
snprintf(lz4Filename, 256, "%s.lz4s-%d", argv[1], BLOCK_BYTES);
snprintf(decFilename, 256, "%s.lz4s-%d.dec", argv[1], BLOCK_BYTES);
printf("inp = [%s]\n", inpFilename);
printf("lz4 = [%s]\n", lz4Filename);
printf("dec = [%s]\n", decFilename);
// compress
{
FILE* inpFp = fopen(inpFilename, "rb");
FILE* outFp = fopen(lz4Filename, "wb");
printf("compress : %s -> %s\n", inpFilename, lz4Filename);
test_compress(outFp, inpFp);
printf("compress : done\n");
fclose(outFp);
fclose(inpFp);
}
// decompress
{
FILE* inpFp = fopen(lz4Filename, "rb");
FILE* outFp = fopen(decFilename, "wb");
printf("decompress : %s -> %s\n", lz4Filename, decFilename);
test_decompress(outFp, inpFp);
printf("decompress : done\n");
fclose(outFp);
fclose(inpFp);
}
// verify
{
FILE* inpFp = fopen(inpFilename, "rb");
FILE* decFp = fopen(decFilename, "rb");
printf("verify : %s <-> %s\n", inpFilename, decFilename);
const int cmp = compare(inpFp, decFp);
if(0 == cmp) {
printf("verify : OK\n");
} else {
printf("verify : NG\n");
}
fclose(decFp);
fclose(inpFp);
}
return 0;
}

100
lz4/examples/blockStreaming_doubleBuffer.md Normal file
View file

@ -0,0 +1,100 @@
# LZ4 Streaming API Example : Double Buffer
by *Takayuki Matsuoka*
`blockStreaming_doubleBuffer.c` is LZ4 Streaming API example which implements double buffer (de)compression.
Please note :
- Firstly, read "LZ4 Streaming API Basics".
- This is relatively advanced application example.
- Output file is not compatible with lz4frame and platform dependent.
## What's the point of this example ?
- Handle huge file in small amount of memory
- Always better compression ratio than Block API
- Uniform block size
## How the compression works
First of all, allocate "Double Buffer" for input and LZ4 compressed data buffer for output.
Double buffer has two pages, "first" page (Page#1) and "second" page (Page#2).
```
Double Buffer
Page#1 Page#2
+---------+---------+
| Block#1 | |
+----+----+---------+
|
v
{Out#1}
Prefix Dependency
+---------+
| |
v |
+---------+----+----+
| Block#1 | Block#2 |
+---------+----+----+
|
v
{Out#2}
External Dictionary Mode
+---------+
| |
| v
+----+----+---------+
| Block#3 | Block#2 |
+----+----+---------+
|
v
{Out#3}
Prefix Dependency
+---------+
| |
v |
+---------+----+----+
| Block#3 | Block#4 |
+---------+----+----+
|
v
{Out#4}
```
Next, read first block to double buffer's first page. And compress it by `LZ4_compress_continue()`.
For the first time, LZ4 doesn't know any previous dependencies,
so it just compress the line without dependencies and generates compressed block {Out#1} to LZ4 compressed data buffer.
After that, write {Out#1} to the file.
Next, read second block to double buffer's second page. And compress it.
This time, LZ4 can use dependency to Block#1 to improve compression ratio.
This dependency is called "Prefix mode".
Next, read third block to double buffer's *first* page, and compress it.
Also this time, LZ4 can use dependency to Block#2.
This dependency is called "External Dictonaly mode".
Continue these procedure to the end of the file.
## How the decompression works
Decompression will do reverse order.
- Read first compressed block.
- Decompress it to the first page and write that page to the file.
- Read second compressed block.
- Decompress it to the second page and write that page to the file.
- Read third compressed block.
- Decompress it to the *first* page and write that page to the file.
Continue these procedure to the end of the compressed file.

211
lz4/examples/blockStreaming_lineByLine.c Normal file
View file

@ -0,0 +1,211 @@
// LZ4 streaming API example : line-by-line logfile compression
// by Takayuki Matsuoka
#if defined(_MSC_VER) && (_MSC_VER <= 1800) /* Visual Studio <= 2013 */
# define _CRT_SECURE_NO_WARNINGS
# define snprintf sprintf_s
#endif
#include "lz4.h"
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
static size_t write_uint16(FILE* fp, uint16_t i)
{
return fwrite(&i, sizeof(i), 1, fp);
}
static size_t write_bin(FILE* fp, const void* array, int arrayBytes)
{
return fwrite(array, 1, arrayBytes, fp);
}
static size_t read_uint16(FILE* fp, uint16_t* i)
{
return fread(i, sizeof(*i), 1, fp);
}
static size_t read_bin(FILE* fp, void* array, int arrayBytes)
{
return fread(array, 1, arrayBytes, fp);
}
static void test_compress(
FILE* outFp,
FILE* inpFp,
size_t messageMaxBytes,
size_t ringBufferBytes)
{
LZ4_stream_t* const lz4Stream = LZ4_createStream();
const size_t cmpBufBytes = LZ4_COMPRESSBOUND(messageMaxBytes);
char* const cmpBuf = (char*) malloc(cmpBufBytes);
char* const inpBuf = (char*) malloc(ringBufferBytes);
int inpOffset = 0;
for ( ; ; )
{
char* const inpPtr = &inpBuf[inpOffset];
#if 0
// Read random length data to the ring buffer.
const int randomLength = (rand() % messageMaxBytes) + 1;
const int inpBytes = (int) read_bin(inpFp, inpPtr, randomLength);
if (0 == inpBytes) break;
#else
// Read line to the ring buffer.
int inpBytes = 0;
if (!fgets(inpPtr, (int) messageMaxBytes, inpFp))
break;
inpBytes = (int) strlen(inpPtr);
#endif
{
const int cmpBytes = LZ4_compress_fast_continue(
lz4Stream, inpPtr, cmpBuf, inpBytes, cmpBufBytes, 1);
if (cmpBytes <= 0) break;
write_uint16(outFp, (uint16_t) cmpBytes);
write_bin(outFp, cmpBuf, cmpBytes);
// Add and wraparound the ringbuffer offset
inpOffset += inpBytes;
if ((size_t)inpOffset >= ringBufferBytes - messageMaxBytes) inpOffset = 0;
}
}
write_uint16(outFp, 0);
free(inpBuf);
free(cmpBuf);
LZ4_freeStream(lz4Stream);
}
static void test_decompress(
FILE* outFp,
FILE* inpFp,
size_t messageMaxBytes,
size_t ringBufferBytes)
{
LZ4_streamDecode_t* const lz4StreamDecode = LZ4_createStreamDecode();
char* const cmpBuf = (char*) malloc(LZ4_COMPRESSBOUND(messageMaxBytes));
char* const decBuf = (char*) malloc(ringBufferBytes);
int decOffset = 0;
for ( ; ; )
{
uint16_t cmpBytes = 0;
if (read_uint16(inpFp, &cmpBytes) != 1) break;
if (cmpBytes == 0) break;
if (read_bin(inpFp, cmpBuf, cmpBytes) != cmpBytes) break;
{
char* const decPtr = &decBuf[decOffset];
const int decBytes = LZ4_decompress_safe_continue(
lz4StreamDecode, cmpBuf, decPtr, cmpBytes, (int) messageMaxBytes);
if (decBytes <= 0) break;
write_bin(outFp, decPtr, decBytes);
// Add and wraparound the ringbuffer offset
decOffset += decBytes;
if ((size_t)decOffset >= ringBufferBytes - messageMaxBytes) decOffset = 0;
}
}
free(decBuf);
free(cmpBuf);
LZ4_freeStreamDecode(lz4StreamDecode);
}
static int compare(FILE* f0, FILE* f1)
{
int result = 0;
const size_t tempBufferBytes = 65536;
char* const b0 = (char*) malloc(tempBufferBytes);
char* const b1 = (char*) malloc(tempBufferBytes);
while(0 == result)
{
const size_t r0 = fread(b0, 1, tempBufferBytes, f0);
const size_t r1 = fread(b1, 1, tempBufferBytes, f1);
result = (int) r0 - (int) r1;
if (0 == r0 || 0 == r1) break;
if (0 == result) result = memcmp(b0, b1, r0);
}
free(b1);
free(b0);
return result;
}
int main(int argc, char* argv[])
{
enum {
MESSAGE_MAX_BYTES = 1024,
RING_BUFFER_BYTES = 1024 * 256 + MESSAGE_MAX_BYTES,
};
char inpFilename[256] = { 0 };
char lz4Filename[256] = { 0 };
char decFilename[256] = { 0 };
if (argc < 2)
{
printf("Please specify input filename\n");
return 0;
}
snprintf(inpFilename, 256, "%s", argv[1]);
snprintf(lz4Filename, 256, "%s.lz4s", argv[1]);
snprintf(decFilename, 256, "%s.lz4s.dec", argv[1]);
printf("inp = [%s]\n", inpFilename);
printf("lz4 = [%s]\n", lz4Filename);
printf("dec = [%s]\n", decFilename);
// compress
{
FILE* inpFp = fopen(inpFilename, "rb");
FILE* outFp = fopen(lz4Filename, "wb");
test_compress(outFp, inpFp, MESSAGE_MAX_BYTES, RING_BUFFER_BYTES);
fclose(outFp);
fclose(inpFp);
}
// decompress
{
FILE* inpFp = fopen(lz4Filename, "rb");
FILE* outFp = fopen(decFilename, "wb");
test_decompress(outFp, inpFp, MESSAGE_MAX_BYTES, RING_BUFFER_BYTES);
fclose(outFp);
fclose(inpFp);
}
// verify
{
FILE* inpFp = fopen(inpFilename, "rb");
FILE* decFp = fopen(decFilename, "rb");
const int cmp = compare(inpFp, decFp);
if (0 == cmp)
printf("Verify : OK\n");
else
printf("Verify : NG\n");
fclose(decFp);
fclose(inpFp);
}
return 0;
}

122
lz4/examples/blockStreaming_lineByLine.md Normal file
View file

@ -0,0 +1,122 @@
# LZ4 Streaming API Example : Line by Line Text Compression
by *Takayuki Matsuoka*
`blockStreaming_lineByLine.c` is LZ4 Straming API example which implements line by line incremental (de)compression.
Please note the following restrictions :
- Firstly, read "LZ4 Streaming API Basics".
- This is relatively advanced application example.
- Output file is not compatible with lz4frame and platform dependent.
## What's the point of this example ?
- Line by line incremental (de)compression.
- Handle huge file in small amount of memory
- Generally better compression ratio than Block API
- Non-uniform block size
## How the compression works
First of all, allocate "Ring Buffer" for input and LZ4 compressed data buffer for output.
```
(1)
Ring Buffer
+--------+
| Line#1 |
+---+----+
|
v
{Out#1}
(2)
Prefix Mode Dependency
+----+
| |
v |
+--------+-+------+
| Line#1 | Line#2 |
+--------+---+----+
|
v
{Out#2}
(3)
Prefix Prefix
+----+ +----+
| | | |
v | v |
+--------+-+------+-+------+
| Line#1 | Line#2 | Line#3 |
+--------+--------+---+----+
|
v
{Out#3}
(4)
External Dictionary Mode
+----+ +----+
| | | |
v | v |
------+--------+-+------+-+--------+
| .... | Line#X | Line#X+1 |
------+--------+--------+-----+----+
^ |
| v
| {Out#X+1}
|
Reset
(5)
Prefix
+-----+
| |
v |
------+--------+--------+----------+--+-------+
| .... | Line#X | Line#X+1 | Line#X+2 |
------+--------+--------+----------+-----+----+
^ |
| v
| {Out#X+2}
|
Reset
```
Next (see (1)), read first line to ringbuffer and compress it by `LZ4_compress_continue()`.
For the first time, LZ4 doesn't know any previous dependencies,
so it just compress the line without dependencies and generates compressed line {Out#1} to LZ4 compressed data buffer.
After that, write {Out#1} to the file and forward ringbuffer offset.
Do the same things to second line (see (2)).
But in this time, LZ4 can use dependency to Line#1 to improve compression ratio.
This dependency is called "Prefix mode".
Eventually, we'll reach end of ringbuffer at Line#X (see (4)).
This time, we should reset ringbuffer offset.
After resetting, at Line#X+1 pointer is not adjacent, but LZ4 still maintain its memory.
This is called "External Dictionary Mode".
In Line#X+2 (see (5)), finally LZ4 forget almost all memories but still remains Line#X+1.
This is the same situation as Line#2.
Continue these procedure to the end of text file.
## How the decompression works
Decompression will do reverse order.
- Read compressed line from the file to buffer.
- Decompress it to the ringbuffer.
- Output decompressed plain text line to the file.
- Forward ringbuffer offset. If offset exceedes end of the ringbuffer, reset it.
Continue these procedure to the end of the compressed file.

190
lz4/examples/blockStreaming_ringBuffer.c Normal file
View file

@ -0,0 +1,190 @@
/* LZ4 streaming API example : ring buffer
* Based on sample code from Takayuki Matsuoka */
/**************************************
* Compiler Options
**************************************/
#if defined(_MSC_VER) && (_MSC_VER <= 1800) /* Visual Studio <= 2013 */
# define _CRT_SECURE_NO_WARNINGS
# define snprintf sprintf_s
#endif
/**************************************
* Includes
**************************************/
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include "lz4.h"
enum {
MESSAGE_MAX_BYTES = 1024,
RING_BUFFER_BYTES = 1024 * 8 + MESSAGE_MAX_BYTES,
DECODE_RING_BUFFER = RING_BUFFER_BYTES + MESSAGE_MAX_BYTES /* Intentionally larger, to test unsynchronized ring buffers */
};
size_t write_int32(FILE* fp, int32_t i) {
return fwrite(&i, sizeof(i), 1, fp);
}
size_t write_bin(FILE* fp, const void* array, int arrayBytes) {
return fwrite(array, 1, arrayBytes, fp);
}
size_t read_int32(FILE* fp, int32_t* i) {
return fread(i, sizeof(*i), 1, fp);
}
size_t read_bin(FILE* fp, void* array, int arrayBytes) {
return fread(array, 1, arrayBytes, fp);
}
void test_compress(FILE* outFp, FILE* inpFp)
{
LZ4_stream_t lz4Stream_body = { { 0 } };
LZ4_stream_t* lz4Stream = &lz4Stream_body;
static char inpBuf[RING_BUFFER_BYTES];
int inpOffset = 0;
for(;;) {
// Read random length ([1,MESSAGE_MAX_BYTES]) data to the ring buffer.
char* const inpPtr = &inpBuf[inpOffset];
const int randomLength = (rand() % MESSAGE_MAX_BYTES) + 1;
const int inpBytes = (int) read_bin(inpFp, inpPtr, randomLength);
if (0 == inpBytes) break;
{
#define CMPBUFSIZE (LZ4_COMPRESSBOUND(MESSAGE_MAX_BYTES))
char cmpBuf[CMPBUFSIZE];
const int cmpBytes = LZ4_compress_fast_continue(lz4Stream, inpPtr, cmpBuf, inpBytes, CMPBUFSIZE, 0);
if(cmpBytes <= 0) break;
write_int32(outFp, cmpBytes);
write_bin(outFp, cmpBuf, cmpBytes);
inpOffset += inpBytes;
// Wraparound the ringbuffer offset
if(inpOffset >= RING_BUFFER_BYTES - MESSAGE_MAX_BYTES) inpOffset = 0;
}
}
write_int32(outFp, 0);
}
void test_decompress(FILE* outFp, FILE* inpFp)
{
static char decBuf[DECODE_RING_BUFFER];
int decOffset = 0;
LZ4_streamDecode_t lz4StreamDecode_body = { { 0 } };
LZ4_streamDecode_t* lz4StreamDecode = &lz4StreamDecode_body;
for(;;) {
int cmpBytes = 0;
char cmpBuf[CMPBUFSIZE];
{ const size_t r0 = read_int32(inpFp, &cmpBytes);
if(r0 != 1 || cmpBytes <= 0) break;
const size_t r1 = read_bin(inpFp, cmpBuf, cmpBytes);
if(r1 != (size_t) cmpBytes) break;
}
{ char* const decPtr = &decBuf[decOffset];
const int decBytes = LZ4_decompress_safe_continue(
lz4StreamDecode, cmpBuf, decPtr, cmpBytes, MESSAGE_MAX_BYTES);
if(decBytes <= 0) break;
decOffset += decBytes;
write_bin(outFp, decPtr, decBytes);
// Wraparound the ringbuffer offset
if(decOffset >= DECODE_RING_BUFFER - MESSAGE_MAX_BYTES) decOffset = 0;
}
}
}
int compare(FILE* f0, FILE* f1)
{
int result = 0;
while (0 == result) {
char b0[65536];
char b1[65536];
const size_t r0 = fread(b0, 1, sizeof(b0), f0);
const size_t r1 = fread(b1, 1, sizeof(b1), f1);
result = (int) r0 - (int) r1;
if (0 == r0 || 0 == r1) break;
if (0 == result) result = memcmp(b0, b1, r0);
}
return result;
}
int main(int argc, char** argv)
{
char inpFilename[256] = { 0 };
char lz4Filename[256] = { 0 };
char decFilename[256] = { 0 };
if (argc < 2) {
printf("Please specify input filename\n");
return 0;
}
snprintf(inpFilename, 256, "%s", argv[1]);
snprintf(lz4Filename, 256, "%s.lz4s-%d", argv[1], 0);
snprintf(decFilename, 256, "%s.lz4s-%d.dec", argv[1], 0);
printf("inp = [%s]\n", inpFilename);
printf("lz4 = [%s]\n", lz4Filename);
printf("dec = [%s]\n", decFilename);
// compress
{ FILE* const inpFp = fopen(inpFilename, "rb");
FILE* const outFp = fopen(lz4Filename, "wb");
test_compress(outFp, inpFp);
fclose(outFp);
fclose(inpFp);
}
// decompress
{ FILE* const inpFp = fopen(lz4Filename, "rb");
FILE* const outFp = fopen(decFilename, "wb");
test_decompress(outFp, inpFp);
fclose(outFp);
fclose(inpFp);
}
// verify
{ FILE* const inpFp = fopen(inpFilename, "rb");
FILE* const decFp = fopen(decFilename, "rb");
const int cmp = compare(inpFp, decFp);
if (0 == cmp) {
printf("Verify : OK\n");
} else {
printf("Verify : NG\n");
}
fclose(decFp);
fclose(inpFp);
}
return 0;
}

363
lz4/examples/compress_functions.c Normal file
View file

@ -0,0 +1,363 @@
/*
* compress_functions.c
* Copyright : Kyle Harper
* License : Follows same licensing as the lz4.c/lz4.h program at any given time. Currently, BSD 2.
* Description: A program to demonstrate the various compression functions involved in when using LZ4_compress_default(). The idea
* is to show how each step in the call stack can be used directly, if desired. There is also some benchmarking for
* each function to demonstrate the (probably lack of) performance difference when jumping the stack.
* (If you're new to lz4, please read simple_buffer.c to understand the fundamentals)
*
* The call stack (before theoretical compiler optimizations) for LZ4_compress_default is as follows:
* LZ4_compress_default
* LZ4_compress_fast
* LZ4_compress_fast_extState
* LZ4_compress_generic
*
* LZ4_compress_default()
* This is the recommended function for compressing data. It will serve as the baseline for comparison.
* LZ4_compress_fast()
* Despite its name, it's not a "fast" version of compression. It simply decides if HEAPMODE is set and either
* allocates memory on the heap for a struct or creates the struct directly on the stack. Stack access is generally
* faster but this function itself isn't giving that advantage, it's just some logic for compile time.
* LZ4_compress_fast_extState()
* This simply accepts all the pointers and values collected thus far and adds logic to determine how
* LZ4_compress_generic should be invoked; specifically: can the source fit into a single pass as determined by
* LZ4_64Klimit.
* LZ4_compress_generic()
* As the name suggests, this is the generic function that ultimately does most of the heavy lifting. Calling this
* directly can help avoid some test cases and branching which might be useful in some implementation-specific
* situations, but you really need to know what you're doing AND what you're asking lz4 to do! You also need a
* wrapper function because this function isn't exposed with lz4.h.
*
* The call stack for decompression functions is shallow. There are 2 options:
* LZ4_decompress_safe || LZ4_decompress_fast
* LZ4_decompress_generic
*
* LZ4_decompress_safe
* This is the recommended function for decompressing data. It is considered safe because the caller specifies
* both the size of the compresssed buffer to read as well as the maximum size of the output (decompressed) buffer
* instead of just the latter.
* LZ4_decompress_fast
* Again, despite its name it's not a "fast" version of decompression. It simply frees the caller of sending the
* size of the compressed buffer (it will simply be read-to-end, hence it's non-safety).
* LZ4_decompress_generic
* This is the generic function that both of the LZ4_decompress_* functions above end up calling. Calling this
* directly is not advised, period. Furthermore, it is a static inline function in lz4.c, so there isn't a symbol
* exposed for anyone using lz4.h to utilize.
*
* Special Note About Decompression:
* Using the LZ4_decompress_safe() function protects against malicious (user) input. If you are using data from a
* trusted source, or if your program is the producer (P) as well as its consumer (C) in a PC or MPMC setup, you can
* safely use the LZ4_decompress_fast function
*/
/* Since lz4 compiles with c99 and not gnu/std99 we need to enable POSIX linking for time.h structs and functions. */
#if __STDC_VERSION__ >= 199901L
#define _XOPEN_SOURCE 600
#else
#define _XOPEN_SOURCE 500
#endif
#define _POSIX_C_SOURCE 199309L
/* Includes, for Power! */
#define LZ4_DISABLE_DEPRECATE_WARNINGS /* LZ4_decompress_fast */
#include "lz4.h"
#include <stdio.h> /* for printf() */
#include <stdlib.h> /* for exit() */
#include <string.h> /* for atoi() memcmp() */
#include <stdint.h> /* for uint_types */
#include <inttypes.h> /* for PRIu64 */
#include <time.h> /* for clock_gettime() */
#include <locale.h> /* for setlocale() */
/* We need to know what one billion is for clock timing. */
#define BILLION 1000000000L
/* Create a crude set of test IDs so we can switch on them later (Can't switch() on a char[] or char*). */
#define ID__LZ4_COMPRESS_DEFAULT 1
#define ID__LZ4_COMPRESS_FAST 2
#define ID__LZ4_COMPRESS_FAST_EXTSTATE 3
#define ID__LZ4_COMPRESS_GENERIC 4
#define ID__LZ4_DECOMPRESS_SAFE 5
#define ID__LZ4_DECOMPRESS_FAST 6
/*
* Easy show-error-and-bail function.
*/
void run_screaming(const char *message, const int code) {
printf("%s\n", message);
exit(code);
}
/*
* Centralize the usage function to keep main cleaner.
*/
void usage(const char *message) {
printf("Usage: ./argPerformanceTesting <iterations>\n");
run_screaming(message, 1);
return;
}
/*
* Runs the benchmark for LZ4_compress_* based on function_id.
*/
uint64_t bench(
const char *known_good_dst,
const int function_id,
const int iterations,
const char *src,
char *dst,
const size_t src_size,
const size_t max_dst_size,
const size_t comp_size
) {
uint64_t time_taken = 0;
int rv = 0;
const int warm_up = 5000;
struct timespec start, end;
const int acceleration = 1;
LZ4_stream_t state;
// Select the right function to perform the benchmark on. We perform 5000 initial loops to warm the cache and ensure that dst
// remains matching to known_good_dst between successive calls.
switch(function_id) {
case ID__LZ4_COMPRESS_DEFAULT:
printf("Starting benchmark for function: LZ4_compress_default()\n");
for(int junk=0; junk<warm_up; junk++)
rv = LZ4_compress_default(src, dst, src_size, max_dst_size);
if (rv < 1)
run_screaming("Couldn't run LZ4_compress_default()... error code received is in exit code.", rv);
if (memcmp(known_good_dst, dst, max_dst_size) != 0)
run_screaming("According to memcmp(), the compressed dst we got doesn't match the known_good_dst... ruh roh.", 1);
clock_gettime(CLOCK_MONOTONIC, &start);
for (int i=1; i<=iterations; i++)
LZ4_compress_default(src, dst, src_size, max_dst_size);
break;
case ID__LZ4_COMPRESS_FAST:
printf("Starting benchmark for function: LZ4_compress_fast()\n");
for(int junk=0; junk<warm_up; junk++)
rv = LZ4_compress_fast(src, dst, src_size, max_dst_size, acceleration);
if (rv < 1)
run_screaming("Couldn't run LZ4_compress_fast()... error code received is in exit code.", rv);
if (memcmp(known_good_dst, dst, max_dst_size) != 0)
run_screaming("According to memcmp(), the compressed dst we got doesn't match the known_good_dst... ruh roh.", 1);
clock_gettime(CLOCK_MONOTONIC, &start);
for (int i=1; i<=iterations; i++)
LZ4_compress_fast(src, dst, src_size, max_dst_size, acceleration);
break;
case ID__LZ4_COMPRESS_FAST_EXTSTATE:
printf("Starting benchmark for function: LZ4_compress_fast_extState()\n");
for(int junk=0; junk<warm_up; junk++)
rv = LZ4_compress_fast_extState(&state, src, dst, src_size, max_dst_size, acceleration);
if (rv < 1)
run_screaming("Couldn't run LZ4_compress_fast_extState()... error code received is in exit code.", rv);
if (memcmp(known_good_dst, dst, max_dst_size) != 0)
run_screaming("According to memcmp(), the compressed dst we got doesn't match the known_good_dst... ruh roh.", 1);
clock_gettime(CLOCK_MONOTONIC, &start);
for (int i=1; i<=iterations; i++)
LZ4_compress_fast_extState(&state, src, dst, src_size, max_dst_size, acceleration);
break;
// Disabled until LZ4_compress_generic() is exposed in the header.
// case ID__LZ4_COMPRESS_GENERIC:
// printf("Starting benchmark for function: LZ4_compress_generic()\n");
// LZ4_resetStream((LZ4_stream_t*)&state);
// for(int junk=0; junk<warm_up; junk++) {
// LZ4_resetStream((LZ4_stream_t*)&state);
// //rv = LZ4_compress_generic_wrapper(&state, src, dst, src_size, max_dst_size, notLimited, byU16, noDict, noDictIssue, acceleration);
// LZ4_compress_generic_wrapper(&state, src, dst, src_size, max_dst_size, acceleration);
// }
// if (rv < 1)
// run_screaming("Couldn't run LZ4_compress_generic()... error code received is in exit code.", rv);
// if (memcmp(known_good_dst, dst, max_dst_size) != 0)
// run_screaming("According to memcmp(), the compressed dst we got doesn't match the known_good_dst... ruh roh.", 1);
// for (int i=1; i<=iterations; i++) {
// LZ4_resetStream((LZ4_stream_t*)&state);
// //LZ4_compress_generic_wrapper(&state, src, dst, src_size, max_dst_size, notLimited, byU16, noDict, noDictIssue, acceleration);
// LZ4_compress_generic_wrapper(&state, src, dst, src_size, max_dst_size, acceleration);
// }
// break;
case ID__LZ4_DECOMPRESS_SAFE:
printf("Starting benchmark for function: LZ4_decompress_safe()\n");
for(int junk=0; junk<warm_up; junk++)
rv = LZ4_decompress_safe(src, dst, comp_size, src_size);
if (rv < 1)
run_screaming("Couldn't run LZ4_decompress_safe()... error code received is in exit code.", rv);
if (memcmp(known_good_dst, dst, src_size) != 0)
run_screaming("According to memcmp(), the compressed dst we got doesn't match the known_good_dst... ruh roh.", 1);
clock_gettime(CLOCK_MONOTONIC, &start);
for (int i=1; i<=iterations; i++)
LZ4_decompress_safe(src, dst, comp_size, src_size);
break;
case ID__LZ4_DECOMPRESS_FAST:
printf("Starting benchmark for function: LZ4_decompress_fast()\n");
for(int junk=0; junk<warm_up; junk++)
rv = LZ4_decompress_fast(src, dst, src_size);
if (rv < 1)
run_screaming("Couldn't run LZ4_decompress_fast()... error code received is in exit code.", rv);
if (memcmp(known_good_dst, dst, src_size) != 0)
run_screaming("According to memcmp(), the compressed dst we got doesn't match the known_good_dst... ruh roh.", 1);
clock_gettime(CLOCK_MONOTONIC, &start);
for (int i=1; i<=iterations; i++)
LZ4_decompress_fast(src, dst, src_size);
break;
default:
run_screaming("The test specified isn't valid. Please check your code.", 1);
break;
}
// Stop timer and return time taken.
clock_gettime(CLOCK_MONOTONIC, &end);
time_taken = BILLION *(end.tv_sec - start.tv_sec) + end.tv_nsec - start.tv_nsec;
return time_taken;
}
/*
* main()
* We will demonstrate the use of each function for simplicity sake. Then we will run 2 suites of benchmarking:
* Test suite A) Uses generic Lorem Ipsum text which should be generally compressible insomuch as basic human text is
* compressible for such a small src_size
* Test Suite B) For the sake of testing, see what results we get if the data is drastically easier to compress. IF there are
* indeed losses and IF more compressible data is faster to process, this will exacerbate the findings.
*/
int main(int argc, char **argv) {
// Get and verify options. There's really only 1: How many iterations to run.
int iterations = 1000000;
if (argc > 1)
iterations = atoi(argv[1]);
if (iterations < 1)
usage("Argument 1 (iterations) must be > 0.");
// First we will create 2 sources (char *) of 2000 bytes each. One normal text, the other highly-compressible text.
const char *src = "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed luctus purus et risus vulputate, et mollis orci ullamcorper. Nulla facilisi. Fusce in ligula sed purus varius aliquet interdum vitae justo. Proin quis diam velit. Nulla varius iaculis auctor. Cras volutpat, justo eu dictum pulvinar, elit sem porttitor metus, et imperdiet metus sapien et ante. Nullam nisi nulla, ornare eu tristique eu, dignissim vitae diam. Nulla sagittis porta libero, a accumsan felis sagittis scelerisque. Integer laoreet eleifend congue. Etiam rhoncus leo vel dolor fermentum, quis luctus nisl iaculis. Praesent a erat sapien. Aliquam semper mi in lorem ultrices ultricies. Lorem ipsum dolor sit amet, consectetur adipiscing elit. In feugiat risus sed enim ultrices, at sodales nulla tristique. Maecenas eget pellentesque justo, sed pellentesque lectus. Fusce sagittis sit amet elit vel varius. Donec sed ligula nec ligula vulputate rutrum sed ut lectus. Etiam congue pharetra leo vitae cursus. Morbi enim ante, porttitor ut varius vel, tincidunt quis justo. Nunc iaculis, risus id ultrices semper, metus est efficitur ligula, vel posuere risus nunc eget purus. Ut lorem turpis, condimentum at sem sed, porta aliquam turpis. In ut sapien a nulla dictum tincidunt quis sit amet lorem. Fusce at est egestas, luctus neque eu, consectetur tortor. Phasellus eleifend ultricies nulla ac lobortis. Morbi maximus quam cursus vehicula iaculis. Maecenas cursus vel justo ut rutrum. Curabitur magna orci, dignissim eget dapibus vitae, finibus id lacus. Praesent rhoncus mattis augue vitae bibendum. Praesent porta mauris non ultrices fermentum. Quisque vulputate ipsum in sodales pulvinar. Aliquam nec mollis felis. Donec vitae augue pulvinar, congue nisl sed, pretium purus. Fusce lobortis mi ac neque scelerisque semper. Pellentesque vel est vitae magna aliquet aliquet. Nam non dolor. Nulla facilisi. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos himenaeos. Morbi ac lacinia felis metus.";
const char *hc_src = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
// Set and derive sizes. Since we're using strings, use strlen() + 1 for \0.
const size_t src_size = strlen(src) + 1;
const size_t max_dst_size = LZ4_compressBound(src_size);
int bytes_returned = 0;
// Now build allocations for the data we'll be playing with.
char *dst = calloc(1, max_dst_size);
char *known_good_dst = calloc(1, max_dst_size);
char *known_good_hc_dst = calloc(1, max_dst_size);
if (dst == NULL || known_good_dst == NULL || known_good_hc_dst == NULL)
run_screaming("Couldn't allocate memory for the destination buffers. Sad :(", 1);
// Create known-good buffers to verify our tests with other functions will produce the same results.
bytes_returned = LZ4_compress_default(src, known_good_dst, src_size, max_dst_size);
if (bytes_returned < 1)
run_screaming("Couldn't create a known-good destination buffer for comparison... this is bad.", 1);
const size_t src_comp_size = bytes_returned;
bytes_returned = LZ4_compress_default(hc_src, known_good_hc_dst, src_size, max_dst_size);
if (bytes_returned < 1)
run_screaming("Couldn't create a known-good (highly compressible) destination buffer for comparison... this is bad.", 1);
const size_t hc_src_comp_size = bytes_returned;
/* LZ4_compress_default() */
// This is the default function so we don't need to demonstrate how to use it. See basics.c if you need more basal information.
/* LZ4_compress_fast() */
// Using this function is identical to LZ4_compress_default except we need to specify an "acceleration" value. Defaults to 1.
memset(dst, 0, max_dst_size);
bytes_returned = LZ4_compress_fast(src, dst, src_size, max_dst_size, 1);
if (bytes_returned < 1)
run_screaming("Failed to compress src using LZ4_compress_fast. echo $? for return code.", bytes_returned);
if (memcmp(dst, known_good_dst, bytes_returned) != 0)
run_screaming("According to memcmp(), the value we got in dst from LZ4_compress_fast doesn't match the known-good value. This is bad.", 1);
/* LZ4_compress_fast_extState() */
// Using this function directly requires that we build an LZ4_stream_t struct ourselves. We do NOT have to reset it ourselves.
memset(dst, 0, max_dst_size);
LZ4_stream_t state;
bytes_returned = LZ4_compress_fast_extState(&state, src, dst, src_size, max_dst_size, 1);
if (bytes_returned < 1)
run_screaming("Failed to compress src using LZ4_compress_fast_extState. echo $? for return code.", bytes_returned);
if (memcmp(dst, known_good_dst, bytes_returned) != 0)
run_screaming("According to memcmp(), the value we got in dst from LZ4_compress_fast_extState doesn't match the known-good value. This is bad.", 1);
/* LZ4_compress_generic */
// When you can exactly control the inputs and options of your LZ4 needs, you can use LZ4_compress_generic and fixed (const)
// values for the enum types such as dictionary and limitations. Any other direct-use is probably a bad idea.
//
// That said, the LZ4_compress_generic() function is 'static inline' and does not have a prototype in lz4.h to expose a symbol
// for it. In other words: we can't access it directly. I don't want to submit a PR that modifies lz4.c/h. Yann and others can
// do that if they feel it's worth expanding this example.
//
// I will, however, leave a skeleton of what would be required to use it directly:
/*
memset(dst, 0, max_dst_size);
// LZ4_stream_t state: is already declared above. We can reuse it BUT we have to reset the stream ourselves between each call.
LZ4_resetStream((LZ4_stream_t *)&state);
// Since src size is small we know the following enums will be used: notLimited (0), byU16 (2), noDict (0), noDictIssue (0).
bytes_returned = LZ4_compress_generic(&state, src, dst, src_size, max_dst_size, notLimited, byU16, noDict, noDictIssue, 1);
if (bytes_returned < 1)
run_screaming("Failed to compress src using LZ4_compress_generic. echo $? for return code.", bytes_returned);
if (memcmp(dst, known_good_dst, bytes_returned) != 0)
run_screaming("According to memcmp(), the value we got in dst from LZ4_compress_generic doesn't match the known-good value. This is bad.", 1);
*/
/* Benchmarking */
/* Now we'll run a few rudimentary benchmarks with each function to demonstrate differences in speed based on the function used.
* Remember, we cannot call LZ4_compress_generic() directly (yet) so it's disabled.
*/
// Suite A - Normal Compressibility
char *dst_d = calloc(1, src_size);
memset(dst, 0, max_dst_size);
printf("\nStarting suite A: Normal compressible text.\n");
uint64_t time_taken__default = bench(known_good_dst, ID__LZ4_COMPRESS_DEFAULT, iterations, src, dst, src_size, max_dst_size, src_comp_size);
uint64_t time_taken__fast = bench(known_good_dst, ID__LZ4_COMPRESS_FAST, iterations, src, dst, src_size, max_dst_size, src_comp_size);
uint64_t time_taken__fast_extstate = bench(known_good_dst, ID__LZ4_COMPRESS_FAST_EXTSTATE, iterations, src, dst, src_size, max_dst_size, src_comp_size);
//uint64_t time_taken__generic = bench(known_good_dst, ID__LZ4_COMPRESS_GENERIC, iterations, src, dst, src_size, max_dst_size, src_comp_size);
uint64_t time_taken__decomp_safe = bench(src, ID__LZ4_DECOMPRESS_SAFE, iterations, known_good_dst, dst_d, src_size, max_dst_size, src_comp_size);
uint64_t time_taken__decomp_fast = bench(src, ID__LZ4_DECOMPRESS_FAST, iterations, known_good_dst, dst_d, src_size, max_dst_size, src_comp_size);
// Suite B - Highly Compressible
memset(dst, 0, max_dst_size);
printf("\nStarting suite B: Highly compressible text.\n");
uint64_t time_taken_hc__default = bench(known_good_hc_dst, ID__LZ4_COMPRESS_DEFAULT, iterations, hc_src, dst, src_size, max_dst_size, hc_src_comp_size);
uint64_t time_taken_hc__fast = bench(known_good_hc_dst, ID__LZ4_COMPRESS_FAST, iterations, hc_src, dst, src_size, max_dst_size, hc_src_comp_size);
uint64_t time_taken_hc__fast_extstate = bench(known_good_hc_dst, ID__LZ4_COMPRESS_FAST_EXTSTATE, iterations, hc_src, dst, src_size, max_dst_size, hc_src_comp_size);
//uint64_t time_taken_hc__generic = bench(known_good_hc_dst, ID__LZ4_COMPRESS_GENERIC, iterations, hc_src, dst, src_size, max_dst_size, hc_src_comp_size);
uint64_t time_taken_hc__decomp_safe = bench(hc_src, ID__LZ4_DECOMPRESS_SAFE, iterations, known_good_hc_dst, dst_d, src_size, max_dst_size, hc_src_comp_size);
uint64_t time_taken_hc__decomp_fast = bench(hc_src, ID__LZ4_DECOMPRESS_FAST, iterations, known_good_hc_dst, dst_d, src_size, max_dst_size, hc_src_comp_size);
// Report and leave.
setlocale(LC_ALL, "");
const char *format = "|%-14s|%-30s|%'14.9f|%'16d|%'14d|%'13.2f%%|\n";
const char *header_format = "|%-14s|%-30s|%14s|%16s|%14s|%14s|\n";
const char *separator = "+--------------+------------------------------+--------------+----------------+--------------+--------------+\n";
printf("\n");
printf("%s", separator);
printf(header_format, "Source", "Function Benchmarked", "Total Seconds", "Iterations/sec", "ns/Iteration", "% of default");
printf("%s", separator);
printf(format, "Normal Text", "LZ4_compress_default()", (double)time_taken__default / BILLION, (int)(iterations / ((double)time_taken__default /BILLION)), (int)time_taken__default / iterations, (double)time_taken__default * 100 / time_taken__default);
printf(format, "Normal Text", "LZ4_compress_fast()", (double)time_taken__fast / BILLION, (int)(iterations / ((double)time_taken__fast /BILLION)), (int)time_taken__fast / iterations, (double)time_taken__fast * 100 / time_taken__default);
printf(format, "Normal Text", "LZ4_compress_fast_extState()", (double)time_taken__fast_extstate / BILLION, (int)(iterations / ((double)time_taken__fast_extstate /BILLION)), (int)time_taken__fast_extstate / iterations, (double)time_taken__fast_extstate * 100 / time_taken__default);
//printf(format, "Normal Text", "LZ4_compress_generic()", (double)time_taken__generic / BILLION, (int)(iterations / ((double)time_taken__generic /BILLION)), (int)time_taken__generic / iterations, (double)time_taken__generic * 100 / time_taken__default);
printf(format, "Normal Text", "LZ4_decompress_safe()", (double)time_taken__decomp_safe / BILLION, (int)(iterations / ((double)time_taken__decomp_safe /BILLION)), (int)time_taken__decomp_safe / iterations, (double)time_taken__decomp_safe * 100 / time_taken__default);
printf(format, "Normal Text", "LZ4_decompress_fast()", (double)time_taken__decomp_fast / BILLION, (int)(iterations / ((double)time_taken__decomp_fast /BILLION)), (int)time_taken__decomp_fast / iterations, (double)time_taken__decomp_fast * 100 / time_taken__default);
printf(header_format, "", "", "", "", "", "");
printf(format, "Compressible", "LZ4_compress_default()", (double)time_taken_hc__default / BILLION, (int)(iterations / ((double)time_taken_hc__default /BILLION)), (int)time_taken_hc__default / iterations, (double)time_taken_hc__default * 100 / time_taken_hc__default);
printf(format, "Compressible", "LZ4_compress_fast()", (double)time_taken_hc__fast / BILLION, (int)(iterations / ((double)time_taken_hc__fast /BILLION)), (int)time_taken_hc__fast / iterations, (double)time_taken_hc__fast * 100 / time_taken_hc__default);
printf(format, "Compressible", "LZ4_compress_fast_extState()", (double)time_taken_hc__fast_extstate / BILLION, (int)(iterations / ((double)time_taken_hc__fast_extstate /BILLION)), (int)time_taken_hc__fast_extstate / iterations, (double)time_taken_hc__fast_extstate * 100 / time_taken_hc__default);
//printf(format, "Compressible", "LZ4_compress_generic()", (double)time_taken_hc__generic / BILLION, (int)(iterations / ((double)time_taken_hc__generic /BILLION)), (int)time_taken_hc__generic / iterations, (double)time_taken_hc__generic * 100 / time_taken_hc__default);
printf(format, "Compressible", "LZ4_decompress_safe()", (double)time_taken_hc__decomp_safe / BILLION, (int)(iterations / ((double)time_taken_hc__decomp_safe /BILLION)), (int)time_taken_hc__decomp_safe / iterations, (double)time_taken_hc__decomp_safe * 100 / time_taken_hc__default);
printf(format, "Compressible", "LZ4_decompress_fast()", (double)time_taken_hc__decomp_fast / BILLION, (int)(iterations / ((double)time_taken_hc__decomp_fast /BILLION)), (int)time_taken_hc__decomp_fast / iterations, (double)time_taken_hc__decomp_fast * 100 / time_taken_hc__default);
printf("%s", separator);
printf("\n");
printf("All done, ran %d iterations per test.\n", iterations);
return 0;
}

280
lz4/examples/dictionaryRandomAccess.c Normal file
View file

@ -0,0 +1,280 @@
// LZ4 API example : Dictionary Random Access
#if defined(_MSC_VER) && (_MSC_VER <= 1800) /* Visual Studio <= 2013 */
# define _CRT_SECURE_NO_WARNINGS
# define snprintf sprintf_s
#endif
#include "lz4.h"
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#define MIN(x, y) ((x) < (y) ? (x) : (y))
enum {
BLOCK_BYTES = 1024, /* 1 KiB of uncompressed data in a block */
DICTIONARY_BYTES = 1024, /* Load a 1 KiB dictionary */
MAX_BLOCKS = 1024 /* For simplicity of implementation */
};
/**
* Magic bytes for this test case.
* This is not a great magic number because it is a common word in ASCII.
* However, it is important to have some versioning system in your format.
*/
const char kTestMagic[] = { 'T', 'E', 'S', 'T' };
void write_int(FILE* fp, int i) {
size_t written = fwrite(&i, sizeof(i), 1, fp);
if (written != 1) { exit(10); }
}
void write_bin(FILE* fp, const void* array, size_t arrayBytes) {
size_t written = fwrite(array, 1, arrayBytes, fp);
if (written != arrayBytes) { exit(11); }
}
void read_int(FILE* fp, int* i) {
size_t read = fread(i, sizeof(*i), 1, fp);
if (read != 1) { exit(12); }
}
size_t read_bin(FILE* fp, void* array, size_t arrayBytes) {
size_t read = fread(array, 1, arrayBytes, fp);
if (ferror(fp)) { exit(12); }
return read;
}
void seek_bin(FILE* fp, long offset, int origin) {
if (fseek(fp, offset, origin)) { exit(14); }
}
void test_compress(FILE* outFp, FILE* inpFp, void *dict, int dictSize)
{
LZ4_stream_t lz4Stream_body;
LZ4_stream_t* lz4Stream = &lz4Stream_body;
char inpBuf[BLOCK_BYTES];
int offsets[MAX_BLOCKS];
int *offsetsEnd = offsets;
LZ4_initStream(lz4Stream, sizeof(*lz4Stream));
/* Write header magic */
write_bin(outFp, kTestMagic, sizeof(kTestMagic));
*offsetsEnd++ = sizeof(kTestMagic);
/* Write compressed data blocks. Each block contains BLOCK_BYTES of plain
data except possibly the last. */
for(;;) {
const int inpBytes = (int) read_bin(inpFp, inpBuf, BLOCK_BYTES);
if(0 == inpBytes) {
break;
}
/* Forget previously compressed data and load the dictionary */
LZ4_loadDict(lz4Stream, dict, dictSize);
{
char cmpBuf[LZ4_COMPRESSBOUND(BLOCK_BYTES)];
const int cmpBytes = LZ4_compress_fast_continue(
lz4Stream, inpBuf, cmpBuf, inpBytes, sizeof(cmpBuf), 1);
if(cmpBytes <= 0) { exit(1); }
write_bin(outFp, cmpBuf, (size_t)cmpBytes);
/* Keep track of the offsets */
*offsetsEnd = *(offsetsEnd - 1) + cmpBytes;
++offsetsEnd;
}
if (offsetsEnd - offsets > MAX_BLOCKS) { exit(2); }
}
/* Write the tailing jump table */
{
int *ptr = offsets;
while (ptr != offsetsEnd) {
write_int(outFp, *ptr++);
}
write_int(outFp, offsetsEnd - offsets);
}
}
void test_decompress(FILE* outFp, FILE* inpFp, void *dict, int dictSize, int offset, int length)
{
LZ4_streamDecode_t lz4StreamDecode_body;
LZ4_streamDecode_t* lz4StreamDecode = &lz4StreamDecode_body;
/* The blocks [currentBlock, endBlock) contain the data we want */
int currentBlock = offset / BLOCK_BYTES;
int endBlock = ((offset + length - 1) / BLOCK_BYTES) + 1;
char decBuf[BLOCK_BYTES];
int offsets[MAX_BLOCKS];
/* Special cases */
if (length == 0) { return; }
/* Read the magic bytes */
{
char magic[sizeof(kTestMagic)];
size_t read = read_bin(inpFp, magic, sizeof(magic));
if (read != sizeof(magic)) { exit(1); }
if (memcmp(kTestMagic, magic, sizeof(magic))) { exit(2); }
}
/* Read the offsets tail */
{
int numOffsets;
int block;
int *offsetsPtr = offsets;
seek_bin(inpFp, -4, SEEK_END);
read_int(inpFp, &numOffsets);
if (numOffsets <= endBlock) { exit(3); }
seek_bin(inpFp, -4 * (numOffsets + 1), SEEK_END);
for (block = 0; block <= endBlock; ++block) {
read_int(inpFp, offsetsPtr++);
}
}
/* Seek to the first block to read */
seek_bin(inpFp, offsets[currentBlock], SEEK_SET);
offset = offset % BLOCK_BYTES;
/* Start decoding */
for(; currentBlock < endBlock; ++currentBlock) {
char cmpBuf[LZ4_COMPRESSBOUND(BLOCK_BYTES)];
/* The difference in offsets is the size of the block */
int cmpBytes = offsets[currentBlock + 1] - offsets[currentBlock];
{
const size_t read = read_bin(inpFp, cmpBuf, (size_t)cmpBytes);
if(read != (size_t)cmpBytes) { exit(4); }
}
/* Load the dictionary */
LZ4_setStreamDecode(lz4StreamDecode, dict, dictSize);
{
const int decBytes = LZ4_decompress_safe_continue(
lz4StreamDecode, cmpBuf, decBuf, cmpBytes, BLOCK_BYTES);
if(decBytes <= 0) { exit(5); }
{
/* Write out the part of the data we care about */
int blockLength = MIN(length, (decBytes - offset));
write_bin(outFp, decBuf + offset, (size_t)blockLength);
offset = 0;
length -= blockLength;
}
}
}
}
int compare(FILE* fp0, FILE* fp1, int length)
{
int result = 0;
while(0 == result) {
char b0[4096];
char b1[4096];
const size_t r0 = read_bin(fp0, b0, MIN(length, (int)sizeof(b0)));
const size_t r1 = read_bin(fp1, b1, MIN(length, (int)sizeof(b1)));
result = (int) r0 - (int) r1;
if(0 == r0 || 0 == r1) {
break;
}
if(0 == result) {
result = memcmp(b0, b1, r0);
}
length -= r0;
}
return result;
}
int main(int argc, char* argv[])
{
char inpFilename[256] = { 0 };
char lz4Filename[256] = { 0 };
char decFilename[256] = { 0 };
char dictFilename[256] = { 0 };
int offset;
int length;
char dict[DICTIONARY_BYTES];
int dictSize;
if(argc < 5) {
printf("Usage: %s input dictionary offset length", argv[0]);
return 0;
}
snprintf(inpFilename, 256, "%s", argv[1]);
snprintf(lz4Filename, 256, "%s.lz4s-%d", argv[1], BLOCK_BYTES);
snprintf(decFilename, 256, "%s.lz4s-%d.dec", argv[1], BLOCK_BYTES);
snprintf(dictFilename, 256, "%s", argv[2]);
offset = atoi(argv[3]);
length = atoi(argv[4]);
printf("inp = [%s]\n", inpFilename);
printf("lz4 = [%s]\n", lz4Filename);
printf("dec = [%s]\n", decFilename);
printf("dict = [%s]\n", dictFilename);
printf("offset = [%d]\n", offset);
printf("length = [%d]\n", length);
/* Load dictionary */
{
FILE* dictFp = fopen(dictFilename, "rb");
dictSize = (int)read_bin(dictFp, dict, DICTIONARY_BYTES);
fclose(dictFp);
}
/* compress */
{
FILE* inpFp = fopen(inpFilename, "rb");
FILE* outFp = fopen(lz4Filename, "wb");
printf("compress : %s -> %s\n", inpFilename, lz4Filename);
test_compress(outFp, inpFp, dict, dictSize);
printf("compress : done\n");
fclose(outFp);
fclose(inpFp);
}
/* decompress */
{
FILE* inpFp = fopen(lz4Filename, "rb");
FILE* outFp = fopen(decFilename, "wb");
printf("decompress : %s -> %s\n", lz4Filename, decFilename);
test_decompress(outFp, inpFp, dict, DICTIONARY_BYTES, offset, length);
printf("decompress : done\n");
fclose(outFp);
fclose(inpFp);
}
/* verify */
{
FILE* inpFp = fopen(inpFilename, "rb");
FILE* decFp = fopen(decFilename, "rb");
seek_bin(inpFp, offset, SEEK_SET);
printf("verify : %s <-> %s\n", inpFilename, decFilename);
const int cmp = compare(inpFp, decFp, length);
if(0 == cmp) {
printf("verify : OK\n");
} else {
printf("verify : NG\n");
}
fclose(decFp);
fclose(inpFp);
}
return 0;
}

67
lz4/examples/dictionaryRandomAccess.md Normal file
View file

@ -0,0 +1,67 @@
# LZ4 API Example : Dictionary Random Access
`dictionaryRandomAccess.c` is LZ4 API example which implements dictionary compression and random access decompression.
Please note that the output file is not compatible with lz4frame and is platform dependent.
## What's the point of this example ?
- Dictionary based compression for homogenous files.
- Random access to compressed blocks.
## How the compression works
Reads the dictionary from a file, and uses it as the history for each block.
This allows each block to be independent, but maintains compression ratio.
```
Dictionary
+
|
v
+---------+
| Block#1 |
+----+----+
|
v
{Out#1}
Dictionary
+
|
v
+---------+
| Block#2 |
+----+----+
|
v
{Out#2}
```
After writing the magic bytes `TEST` and then the compressed blocks, write out the jump table.
The last 4 bytes is an integer containing the number of blocks in the stream.
If there are `N` blocks, then just before the last 4 bytes is `N + 1` 4 byte integers containing the offsets at the beginning and end of each block.
Let `Offset#K` be the total number of bytes written after writing out `Block#K` *including* the magic bytes for simplicity.
```
+------+---------+ +---------+---+----------+ +----------+-----+
| TEST | Block#1 | ... | Block#N | 4 | Offset#1 | ... | Offset#N | N+1 |
+------+---------+ +---------+---+----------+ +----------+-----+
```
## How the decompression works
Decompression will do reverse order.
- Seek to the last 4 bytes of the file and read the number of offsets.
- Read each offset into an array.
- Seek to the first block containing data we want to read.
We know where to look because we know each block contains a fixed amount of uncompressed data, except possibly the last.
- Decompress it and write what data we need from it to the file.
- Read the next block.
- Decompress it and write that page to the file.
Continue these procedure until all the required data has been read.

401
lz4/examples/frameCompress.c Normal file
View file

@ -0,0 +1,401 @@
/* LZ4frame API example : compress a file
* Modified from an example code by Zbigniew Jędrzejewski-Szmek
*
* This example streams an input file into an output file
* using a bounded memory budget.
* Input is read in chunks of IN_CHUNK_SIZE */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <assert.h>
#include <lz4frame.h>
#define IN_CHUNK_SIZE (16*1024)
static const LZ4F_preferences_t kPrefs = {
{ LZ4F_max256KB, LZ4F_blockLinked, LZ4F_noContentChecksum, LZ4F_frame,
0 /* unknown content size */, 0 /* no dictID */ , LZ4F_noBlockChecksum },
0, /* compression level; 0 == default */
0, /* autoflush */
0, /* favor decompression speed */
{ 0, 0, 0 }, /* reserved, must be set to 0 */
};
/* safe_fwrite() :
* performs fwrite(), ensure operation success, or immediately exit() */
static void safe_fwrite(void* buf, size_t eltSize, size_t nbElt, FILE* f)
{
size_t const writtenSize = fwrite(buf, eltSize, nbElt, f);
size_t const expectedSize = eltSize * nbElt;
if (nbElt>0) assert(expectedSize / nbElt == eltSize); /* check overflow */
if (writtenSize < expectedSize) {
if (ferror(f)) /* note : ferror() must follow fwrite */
fprintf(stderr, "Write failed \n");
else
fprintf(stderr, "Write too short \n");
exit(1);
}
}
/* ================================================= */
/* Streaming Compression example */
/* ================================================= */
typedef struct {
int error;
unsigned long long size_in;
unsigned long long size_out;
} compressResult_t;
static compressResult_t
compress_file_internal(FILE* f_in, FILE* f_out,
LZ4F_compressionContext_t ctx,
void* inBuff, size_t inChunkSize,
void* outBuff, size_t outCapacity)
{
compressResult_t result = { 1, 0, 0 }; /* result for an error */
unsigned long long count_in = 0, count_out;
assert(f_in != NULL); assert(f_out != NULL);
assert(ctx != NULL);
assert(outCapacity >= LZ4F_HEADER_SIZE_MAX);
assert(outCapacity >= LZ4F_compressBound(inChunkSize, &kPrefs));
/* write frame header */
{ size_t const headerSize = LZ4F_compressBegin(ctx, outBuff, outCapacity, &kPrefs);
if (LZ4F_isError(headerSize)) {
printf("Failed to start compression: error %u \n", (unsigned)headerSize);
return result;
}
count_out = headerSize;
printf("Buffer size is %u bytes, header size %u bytes \n",
(unsigned)outCapacity, (unsigned)headerSize);
safe_fwrite(outBuff, 1, headerSize, f_out);
}
/* stream file */
for (;;) {
size_t const readSize = fread(inBuff, 1, IN_CHUNK_SIZE, f_in);
if (readSize == 0) break; /* nothing left to read from input file */
count_in += readSize;
size_t const compressedSize = LZ4F_compressUpdate(ctx,
outBuff, outCapacity,
inBuff, readSize,
NULL);
if (LZ4F_isError(compressedSize)) {
printf("Compression failed: error %u \n", (unsigned)compressedSize);
return result;
}
printf("Writing %u bytes\n", (unsigned)compressedSize);
safe_fwrite(outBuff, 1, compressedSize, f_out);
count_out += compressedSize;
}
/* flush whatever remains within internal buffers */
{ size_t const compressedSize = LZ4F_compressEnd(ctx,
outBuff, outCapacity,
NULL);
if (LZ4F_isError(compressedSize)) {
printf("Failed to end compression: error %u \n", (unsigned)compressedSize);
return result;
}
printf("Writing %u bytes \n", (unsigned)compressedSize);
safe_fwrite(outBuff, 1, compressedSize, f_out);
count_out += compressedSize;
}
result.size_in = count_in;
result.size_out = count_out;
result.error = 0;
return result;
}
static compressResult_t
compress_file(FILE* f_in, FILE* f_out)
{
assert(f_in != NULL);
assert(f_out != NULL);
/* ressource allocation */
LZ4F_compressionContext_t ctx;
size_t const ctxCreation = LZ4F_createCompressionContext(&ctx, LZ4F_VERSION);
void* const src = malloc(IN_CHUNK_SIZE);
size_t const outbufCapacity = LZ4F_compressBound(IN_CHUNK_SIZE, &kPrefs); /* large enough for any input <= IN_CHUNK_SIZE */
void* const outbuff = malloc(outbufCapacity);
compressResult_t result = { 1, 0, 0 }; /* == error (default) */
if (!LZ4F_isError(ctxCreation) && src && outbuff) {
result = compress_file_internal(f_in, f_out,
ctx,
src, IN_CHUNK_SIZE,
outbuff, outbufCapacity);
} else {
printf("error : ressource allocation failed \n");
}
LZ4F_freeCompressionContext(ctx); /* supports free on NULL */
free(src);
free(outbuff);
return result;
}
/* ================================================= */
/* Streaming decompression example */
/* ================================================= */
static size_t get_block_size(const LZ4F_frameInfo_t* info) {
switch (info->blockSizeID) {
case LZ4F_default:
case LZ4F_max64KB: return 1 << 16;
case LZ4F_max256KB: return 1 << 18;
case LZ4F_max1MB: return 1 << 20;
case LZ4F_max4MB: return 1 << 22;
default:
printf("Impossible with expected frame specification (<=v1.6.1)\n");
exit(1);
}
}
/* @return : 1==error, 0==success */
static int
decompress_file_internal(FILE* f_in, FILE* f_out,
LZ4F_dctx* dctx,
void* src, size_t srcCapacity, size_t filled, size_t alreadyConsumed,
void* dst, size_t dstCapacity)
{
int firstChunk = 1;
size_t ret = 1;
assert(f_in != NULL); assert(f_out != NULL);
assert(dctx != NULL);
assert(src != NULL); assert(srcCapacity > 0); assert(filled <= srcCapacity); assert(alreadyConsumed <= filled);
assert(dst != NULL); assert(dstCapacity > 0);
/* Decompression */
while (ret != 0) {
/* Load more input */
size_t readSize = firstChunk ? filled : fread(src, 1, srcCapacity, f_in); firstChunk=0;
const void* srcPtr = (const char*)src + alreadyConsumed; alreadyConsumed=0;
const void* const srcEnd = (const char*)srcPtr + readSize;
if (readSize == 0 || ferror(f_in)) {
printf("Decompress: not enough input or error reading file\n");
return 1;
}
/* Decompress:
* Continue while there is more input to read (srcPtr != srcEnd)
* and the frame isn't over (ret != 0)
*/
while (srcPtr < srcEnd && ret != 0) {
/* Any data within dst has been flushed at this stage */
size_t dstSize = dstCapacity;
size_t srcSize = (const char*)srcEnd - (const char*)srcPtr;
ret = LZ4F_decompress(dctx, dst, &dstSize, srcPtr, &srcSize, /* LZ4F_decompressOptions_t */ NULL);
if (LZ4F_isError(ret)) {
printf("Decompression error: %s\n", LZ4F_getErrorName(ret));
return 1;
}
/* Flush output */
if (dstSize != 0) safe_fwrite(dst, 1, dstSize, f_out);
/* Update input */
srcPtr = (const char*)srcPtr + srcSize;
}
assert(srcPtr <= srcEnd);
/* Ensure all input data has been consumed.
* It is valid to have multiple frames in the same file,
* but this example only supports one frame.
*/
if (srcPtr < srcEnd) {
printf("Decompress: Trailing data left in file after frame\n");
return 1;
}
}
/* Check that there isn't trailing data in the file after the frame.
* It is valid to have multiple frames in the same file,
* but this example only supports one frame.
*/
{ size_t const readSize = fread(src, 1, 1, f_in);
if (readSize != 0 || !feof(f_in)) {
printf("Decompress: Trailing data left in file after frame\n");
return 1;
} }
return 0;
}
/* @return : 1==error, 0==completed */
static int
decompress_file_allocDst(FILE* f_in, FILE* f_out,
LZ4F_dctx* dctx,
void* src, size_t srcCapacity)
{
assert(f_in != NULL); assert(f_out != NULL);
assert(dctx != NULL);
assert(src != NULL);
assert(srcCapacity >= LZ4F_HEADER_SIZE_MAX); /* ensure LZ4F_getFrameInfo() can read enough data */
/* Read Frame header */
size_t const readSize = fread(src, 1, srcCapacity, f_in);
if (readSize == 0 || ferror(f_in)) {
printf("Decompress: not enough input or error reading file\n");
return 1;
}
LZ4F_frameInfo_t info;
size_t consumedSize = readSize;
{ size_t const fires = LZ4F_getFrameInfo(dctx, &info, src, &consumedSize);
if (LZ4F_isError(fires)) {
printf("LZ4F_getFrameInfo error: %s\n", LZ4F_getErrorName(fires));
return 1;
} }
/* Allocating enough space for an entire block isn't necessary for
* correctness, but it allows some memcpy's to be elided.
*/
size_t const dstCapacity = get_block_size(&info);
void* const dst = malloc(dstCapacity);
if (!dst) { perror("decompress_file(dst)"); return 1; }
int const decompressionResult = decompress_file_internal(
f_in, f_out,
dctx,
src, srcCapacity, readSize-consumedSize, consumedSize,
dst, dstCapacity);
free(dst);
return decompressionResult;
}
/* @result : 1==error, 0==success */
static int decompress_file(FILE* f_in, FILE* f_out)
{
assert(f_in != NULL); assert(f_out != NULL);
/* Ressource allocation */
void* const src = malloc(IN_CHUNK_SIZE);
if (!src) { perror("decompress_file(src)"); return 1; }
LZ4F_dctx* dctx;
{ size_t const dctxStatus = LZ4F_createDecompressionContext(&dctx, LZ4F_VERSION);
if (LZ4F_isError(dctxStatus)) {
printf("LZ4F_dctx creation error: %s\n", LZ4F_getErrorName(dctxStatus));
} }
int const result = !dctx ? 1 /* error */ :
decompress_file_allocDst(f_in, f_out, dctx, src, IN_CHUNK_SIZE);
free(src);
LZ4F_freeDecompressionContext(dctx); /* note : free works on NULL */
return result;
}
int compareFiles(FILE* fp0, FILE* fp1)
{
int result = 0;
while (result==0) {
char b0[1024];
char b1[1024];
size_t const r0 = fread(b0, 1, sizeof(b0), fp0);
size_t const r1 = fread(b1, 1, sizeof(b1), fp1);
result = (r0 != r1);
if (!r0 || !r1) break;
if (!result) result = memcmp(b0, b1, r0);
}
return result;
}
int main(int argc, const char **argv) {
char inpFilename[256] = { 0 };
char lz4Filename[256] = { 0 };
char decFilename[256] = { 0 };
if (argc < 2) {
printf("Please specify input filename\n");
return 0;
}
snprintf(inpFilename, 256, "%s", argv[1]);
snprintf(lz4Filename, 256, "%s.lz4", argv[1]);
snprintf(decFilename, 256, "%s.lz4.dec", argv[1]);
printf("inp = [%s]\n", inpFilename);
printf("lz4 = [%s]\n", lz4Filename);
printf("dec = [%s]\n", decFilename);
/* compress */
{ FILE* const inpFp = fopen(inpFilename, "rb");
FILE* const outFp = fopen(lz4Filename, "wb");
printf("compress : %s -> %s\n", inpFilename, lz4Filename);
compressResult_t const ret = compress_file(inpFp, outFp);
fclose(outFp);
fclose(inpFp);
if (ret.error) {
printf("compress : failed with code %i\n", ret.error);
return ret.error;
}
printf("%s: %zu → %zu bytes, %.1f%%\n",
inpFilename,
(size_t)ret.size_in, (size_t)ret.size_out, /* might overflow is size_t is 32 bits and size_{in,out} > 4 GB */
(double)ret.size_out / ret.size_in * 100);
printf("compress : done\n");
}
/* decompress */
{ FILE* const inpFp = fopen(lz4Filename, "rb");
FILE* const outFp = fopen(decFilename, "wb");
printf("decompress : %s -> %s\n", lz4Filename, decFilename);
int const ret = decompress_file(inpFp, outFp);
fclose(outFp);
fclose(inpFp);
if (ret) {
printf("decompress : failed with code %i\n", ret);
return ret;
}
printf("decompress : done\n");
}
/* verify */
{ FILE* const inpFp = fopen(inpFilename, "rb");
FILE* const decFp = fopen(decFilename, "rb");
printf("verify : %s <-> %s\n", inpFilename, decFilename);
int const cmp = compareFiles(inpFp, decFp);
fclose(decFp);
fclose(inpFp);
if (cmp) {
printf("corruption detected : decompressed file differs from original\n");
return cmp;
}
printf("verify : OK\n");
}
return 0;
}

13
lz4/examples/printVersion.c Normal file
View file

@ -0,0 +1,13 @@
// LZ4 trivial example : print Library version number
// by Takayuki Matsuoka
#include <stdio.h>
#include "lz4.h"
int main(int argc, char** argv)
{
(void)argc; (void)argv;
printf("Hello World ! LZ4 Library version = %d\n", LZ4_versionNumber());
return 0;
}

99
lz4/examples/simple_buffer.c Normal file
View file

@ -0,0 +1,99 @@
/*
* simple_buffer.c
* Copyright : Kyle Harper
* License : Follows same licensing as the lz4.c/lz4.h program at any given time. Currently, BSD 2.
* Description: Example program to demonstrate the basic usage of the compress/decompress functions within lz4.c/lz4.h.
* The functions you'll likely want are LZ4_compress_default and LZ4_decompress_safe.
* Both of these are documented in the lz4.h header file; I recommend reading them.
*/
/* Dependencies */
#include <stdio.h> // For printf()
#include <string.h> // For memcmp()
#include <stdlib.h> // For exit()
#include "lz4.h" // This is all that is required to expose the prototypes for basic compression and decompression.
/*
* Simple show-error-and-bail function.
*/
void run_screaming(const char* message, const int code) {
printf("%s \n", message);
exit(code);
}
/*
* main
*/
int main(void) {
/* Introduction */
// Below we will have a Compression and Decompression section to demonstrate.
// There are a few important notes before we start:
// 1) The return codes of LZ4_ functions are important.
// Read lz4.h if you're unsure what a given code means.
// 2) LZ4 uses char* pointers in all LZ4_ functions.
// This is baked into the API and not going to change, for consistency.
// If your program uses different pointer types,
// you may need to do some casting or set the right -Wno compiler flags to ignore those warnings (e.g.: -Wno-pointer-sign).
/* Compression */
// We'll store some text into a variable pointed to by *src to be compressed later.
const char* const src = "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Lorem ipsum dolor site amat.";
// The compression function needs to know how many bytes exist. Since we're using a string, we can use strlen() + 1 (for \0).
const int src_size = (int)(strlen(src) + 1);
// LZ4 provides a function that will tell you the maximum size of compressed output based on input data via LZ4_compressBound().
const int max_dst_size = LZ4_compressBound(src_size);
// We will use that size for our destination boundary when allocating space.
char* compressed_data = malloc((size_t)max_dst_size);
if (compressed_data == NULL)
run_screaming("Failed to allocate memory for *compressed_data.", 1);
// That's all the information and preparation LZ4 needs to compress *src into *compressed_data.
// Invoke LZ4_compress_default now with our size values and pointers to our memory locations.
// Save the return value for error checking.
const int compressed_data_size = LZ4_compress_default(src, compressed_data, src_size, max_dst_size);
// Check return_value to determine what happened.
if (compressed_data_size <= 0)
run_screaming("A 0 or negative result from LZ4_compress_default() indicates a failure trying to compress the data. ", 1);
if (compressed_data_size > 0)
printf("We successfully compressed some data! Ratio: %.2f\n",
(float) compressed_data_size/src_size);
// Not only does a positive return_value mean success, the value returned == the number of bytes required.
// You can use this to realloc() *compress_data to free up memory, if desired. We'll do so just to demonstrate the concept.
compressed_data = (char *)realloc(compressed_data, (size_t)compressed_data_size);
if (compressed_data == NULL)
run_screaming("Failed to re-alloc memory for compressed_data. Sad :(", 1);
/* Decompression */
// Now that we've successfully compressed the information from *src to *compressed_data, let's do the opposite!
// The decompression will need to know the compressed size, and an upper bound of the decompressed size.
// In this example, we just re-use this information from previous section,
// but in a real-world scenario, metadata must be transmitted to the decompression side.
// Each implementation is in charge of this part. Oftentimes, it adds some header of its own.
// Sometimes, the metadata can be extracted from the local context.
// First, let's create a *new_src location of size src_size since we know that value.
char* const regen_buffer = malloc(src_size);
if (regen_buffer == NULL)
run_screaming("Failed to allocate memory for *regen_buffer.", 1);
// The LZ4_decompress_safe function needs to know where the compressed data is, how many bytes long it is,
// where the regen_buffer memory location is, and how large regen_buffer (uncompressed) output will be.
// Again, save the return_value.
const int decompressed_size = LZ4_decompress_safe(compressed_data, regen_buffer, compressed_data_size, src_size);
free(compressed_data); /* no longer useful */
if (decompressed_size < 0)
run_screaming("A negative result from LZ4_decompress_safe indicates a failure trying to decompress the data. See exit code (echo $?) for value returned.", decompressed_size);
if (decompressed_size >= 0)
printf("We successfully decompressed some data!\n");
// Not only does a positive return value mean success,
// value returned == number of bytes regenerated from compressed_data stream.
if (decompressed_size != src_size)
run_screaming("Decompressed data is different from original! \n", 1);
/* Validation */
// We should be able to compare our original *src with our *new_src and be byte-for-byte identical.
if (memcmp(src, regen_buffer, src_size) != 0)
run_screaming("Validation failed. *src and *new_src are not identical.", 1);
printf("Validation done. The string we ended up with is:\n%s\n", regen_buffer);
return 0;
}

87
lz4/examples/streaming_api_basics.md Normal file
View file

@ -0,0 +1,87 @@
# LZ4 Streaming API Basics
by *Takayuki Matsuoka*
## LZ4 API sets
LZ4 has the following API sets :
- "Auto Framing" API (lz4frame.h) :
This is most recommended API for usual application.
It guarantees interoperability with other LZ4 framing format compliant tools/libraries
such as LZ4 command line utility, node-lz4, etc.
- "Block" API : This is recommended for simple purpose.
It compress single raw memory block to LZ4 memory block and vice versa.
- "Streaming" API : This is designed for complex things.
For example, compress huge stream data in restricted memory environment.
Basically, you should use "Auto Framing" API.
But if you want to write advanced application, it's time to use Block or Streaming APIs.
## What is difference between Block and Streaming API ?
Block API (de)compresses a single contiguous memory block.
In other words, LZ4 library finds redundancy from a single contiguous memory block.
Streaming API does same thing but (de)compresses multiple adjacent contiguous memory blocks.
So LZ4 library could find more redundancy than Block API.
The following figure shows difference between API and block sizes.
In these figures, the original data is split into 4KiBytes contiguous chunks.
```
Original Data
+---------------+---------------+----+----+----+
| 4KiB Chunk A | 4KiB Chunk B | C | D |... |
+---------------+---------------+----+----+----+
Example (1) : Block API, 4KiB Block
+---------------+---------------+----+----+----+
| 4KiB Chunk A | 4KiB Chunk B | C | D |... |
+---------------+---------------+----+----+----+
| Block #1 | Block #2 | #3 | #4 |... |
+---------------+---------------+----+----+----+
(No Dependency)
Example (2) : Block API, 8KiB Block
+---------------+---------------+----+----+----+
| 4KiB Chunk A | 4KiB Chunk B | C | D |... |
+---------------+---------------+----+----+----+
| Block #1 |Block #2 |... |
+--------------------+----------+-------+-+----+
^ | ^ |
| | | |
+--------------+ +----+
Internal Dependency Internal Dependency
Example (3) : Streaming API, 4KiB Block
+---------------+---------------+-----+----+----+
| 4KiB Chunk A | 4KiB Chunk B | C | D |... |
+---------------+---------------+-----+----+----+
| Block #1 | Block #2 | #3 | #4 |... |
+---------------+----+----------+-+---+-+--+----+
^ | ^ | ^ |
| | | | | |
+--------------+ +--------+ +---+
Dependency Dependency Dependency
```
- In example (1), there is no dependency.
All blocks are compressed independently.
- In example (2), naturally 8KiBytes block has internal dependency.
But still block #1 and #2 are compressed independently.
- In example (3), block #2 has dependency to #1,
also #3 has dependency to #2 and #1, #4 has #3, #2 and #1, and so on.
Here, we can observe difference between example (2) and (3).
In (2), there's no dependency between chunk B and C, but (3) has dependency between B and C.
This dependency improves compression ratio.
## Restriction of Streaming API
For efficiency, Streaming API doesn't keep a mirror copy of dependent (de)compressed memory.
This means users should keep these dependent (de)compressed memory explicitly.
Usually, "Dependent memory" is previous adjacent contiguous memory up to 64KiBytes.
LZ4 will not access further memories.