Google Open-sources JPEG compression module

BlitzMax Forums/Brucey's Modules/Google Open-sources JPEG compression module

xlsior

(Posted March) [#1]

Looks like google just open-sourced their Guetzli JPEG encoding source code under the Apache license, which claims 35% smaller filesizes at comparable visuals to typical jpeg encoding. (Best results on source images that aren't already jpeg compressed)

https://betanews.com/2017/03/16/google-guetzli-open-source-jpeg-encoder/

Brucey: Any chance you'd want to port this one to Blitzmax?

Grisu

(Posted March) [#2]

Is this new format readable by older apps?

xlsior

(Posted March) [#3]

Is this new format readable by older apps?

Yes -- it's NOT a new format, it's same old JPEG, except it is somewhat content-aware and tries a bunch of different methods of compressing parts of the image to automatically pick the most efficient solutions for different parts of the image instead of blindly using the same steps throughout.

The end result is a jpeg that's on average 35% smaller than traditional, but the resulting file is still 100% jpeg, and any 20 year old JPEG viewer won't have any issues opening it.

The downsides are that it is quite a bit slower than traditional JPEG savers, and that it takes significantly more memory while analyzing the source image.

BlitzSupport

(Posted March) [#4]

I'd kind of skipped trying this due to believing it would need a new decoder!

I've tried converting a couple of test JPGs -- makes a big size difference on a larger file, and I certainly can't see any difference.

Just tried a converted JPG with BlitzMax and it loaded just fine.

Does take *forever* to convert, but hey-ho... impressive results.

(For anyone who wants to quickly test, jump to the Github Releases page for a pre-built command-line executable -- parameters are simply in-file and out-file.)

Grisu

(Posted March) [#5]

Is there any size difference in comparison to a small png files (~272x85 pixels)?

The older JPG-format was larger and lost a lot of quailty too. I'm looking for an alternative for my PRP app.

Brucey

(Posted March) [#6]

I've just committed BaH.Guetzli.

I have used the same SavePixmapJpeg() API as the BRL JPEG module...

Function SavePixmapJPeg:Int( pixmap:TPixmap, url:Object, quality:Int = DEFAULT_JPEQ_QUALITY )

So far only tested on MacOS (64-bit NG & 10.9 min version) and Windows (64-bit NG with MinGWw64 6.2.0)

You are likely to want to use this in 64-bit given the rather sumptuous memory requirements for reasonably image sizes.
Otherwise, YMMV ;-)

xlsior

(Posted March) [#7]

Cool, thanks!

Grisu

(Posted March) [#8]

Awesome. Thanks Brucey.

xlsior

(Posted March) [#9]

So far only tested on MacOS (64-bit NG & 10.9 min version) and Windows (64-bit NG with MinGWw64 6.2.0)

Which MinGW distribution are you using under windows now?

I thought you used to use TDM, but the most recent version of that still appears to be 5.1.0? Do you have a download link somewhere for the distribution you are using?

Brucey

(Posted March) [#10]

I thought you used to use TDM, but the most recent version of that still appears to be 5.1.0

I generally do, but I've recently been testing MinGW-w64. The only issue with that is that they don't ship a combined 32/64-bit build system - you need to have both the 32-bit and 64-bit packages on your box (if you want to build for both targets)... which the latest BMK now supports. (You can now have BlitzMax/MinGW32x86 and BlitzMax/MinGW32x64 specific folders, or BlitzMax/MinGW32 for both)

I got mine from here : https://sourceforge.net/projects/mingw-w64/files/
If you look under the main list, you'll see individual sets of downloads for different variations of MinGW-W64.
I don't have the specific ones I used at the moment. Will update here later.

Grisu

(Posted March) [#11]

Is it possible to upgrade bmx vanilla with the gcc 6.3 win32 builds too?

braxton

(Posted March) [#12]

Hi Brucey,

Thank you, you never cease to amaze me, I am still in the process of learning Git so that I can try and submit a few module fixes for NG, since I have moved everything over to Windows x64.

@Grisu,

Thanks to the the January 2017 GitHub commit to bmx vanilla by Simon Armstrong, you can now use the following version, which I now use to build NG's bcc and bmk and it works really well

i686-6.3.0-release-posix-dwarf-rt_v5-rev1

Grisu

(Posted March) [#13]

@braxton
Couldy you help me to set it up properly?

I download BMX from here: https://github.com/nitrologic/blitzmax
And GCC 6.3 from here: https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Personal%20Builds/mingw-builds/6.3.0/threads-win32/dwarf/i686-6.3.0-release-win32-dwarf-rt_v5-rev1.7z/download

I created a fresh MinGW32 folder (before I was using TDM 5.1.0).

When I recompile all modules, I get an error that I'm missing a "libintl-8.dll":

Sorry for the hijacking this thread in advance. :(

|Edit|
When I copy the ar.exe and ld.exe over to the bmx folder, I can rebuild all modules fine. Though when I run a source code, I still get errors:

C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x3e): undefined reference to `mingw_initcharmax'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x4e): undefined reference to `mingw_app_type'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x6a): undefined reference to `_encode_pointer'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x75): undefined reference to `__onexitend'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x7a): undefined reference to `__onexitbegin'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x86): undefined reference to `_setargv'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x8c): undefined reference to `_MINGW_INSTALL_DEBUG_MATHERR'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x103): undefined reference to `_matherr'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x108): undefined reference to `__mingw_setusermatherr'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x134): undefined reference to `_newmode'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x15d): undefined reference to `_dowildcard'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x1d6): undefined reference to `mingw_app_type'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x20f): undefined reference to `__native_startup_lock'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x218): undefined reference to `__native_startup_state'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x228): undefined reference to `__native_startup_state'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x23f): undefined reference to `__native_startup_state'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x281): undefined reference to `_gnu_exception_handler@4'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x28f): undefined reference to `__mingw_oldexcpt_handler'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x29b): undefined reference to `_set_invalid_parameter_handler'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x316): undefined reference to `mingw_app_type'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x413): undefined reference to `__native_startup_state'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x432): undefined reference to `__native_startup_state'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x443): undefined reference to `__xc_z'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x44a): undefined reference to `__xc_a'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x457): undefined reference to `__native_startup_state'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x467): undefined reference to `__native_startup_lock'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x483): undefined reference to `__native_startup_state'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x48f): undefined reference to `__xi_z'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x496): undefined reference to `__xi_a'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x4c5): undefined reference to `mingw_app_type'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x4ce): undefined reference to `__security_init_cookie'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x4e5): undefined reference to `mingw_app_type'
C:/MinGW32/bin/../lib/gcc/x86_64-w64-mingw32/6.3.0/../../../../x86_64-w64-mingw32/lib/../lib32/crt2.o:crtexe.c:(.text+0x4ee): undefined reference to `__security_init_cookie'
C:/BlitzMax/mod/srs.mod/fmodencoder2.mod/fmodencoder2.release.win32.x86.a(FModEncoder.cpp.release.win32.x86.o):FModEncoder.cpp:(.text+0x21): undefined reference to `operator delete(void*, unsigned int)'
C:/BlitzMax/mod/srs.mod/fmodencoder2.mod/fmodencoder2.release.win32.x86.a(FModEncoder.cpp.release.win32.x86.o):FModEncoder.cpp:(.text+0x115): undefined reference to `operator delete(void*, unsigned int)'
C:/BlitzMax/mod/srs.mod/fmodencoder2.mod/fmodencoder2.release.win32.x86.a(FModEncoder.cpp.release.win32.x86.o):FModEncoder.cpp:(.text+0x181): undefined reference to `operator delete(void*, unsigned int)'
C:/BlitzMax/mod/srs.mod/fmodencoder2.mod/fmodencoder2.release.win32.x86.a(FModEncoder.cpp.release.win32.x86.o):FModEncoder.cpp:(.text+0x323): undefined reference to `operator delete(void*, unsigned int)'
C:/BlitzMax/mod/bah.mod/libcurl.mod/libcurl.release.win32.x86.a(curl_glue.cpp.release.win32.x86.o):curl_glue.cpp:(.text+0x5a2): undefined reference to `operator delete(void*, unsigned int)'
C:/BlitzMax/mod/bah.mod/libcurl.mod/libcurl.release.win32.x86.a(curl_glue.cpp.release.win32.x86.o):curl_glue.cpp:(.text+0x8b2): more undefined references to `operator delete(void*, unsigned int)' follow
C:/BlitzMax/mod/bah.mod/libcurl.mod/libcurl.release.win32.x86.a(http.c.release.win32.x86.o):http.c:(.text+0x4e25): undefined reference to `_time32'
C:/BlitzMax/mod/bah.mod/libcurl.mod/libcurl.release.win32.x86.a(hostip.c.release.win32.x86.o):hostip.c:(.text+0xee): undefined reference to `_time32'
C:/BlitzMax/mod/bah.mod/libcurl.mod/libcurl.release.win32.x86.a(hostip.c.release.win32.x86.o):hostip.c:(.text+0x28f): undefined reference to `_time32'
C:/BlitzMax/mod/bah.mod/libcurl.mod/libcurl.release.win32.x86.a(hostip.c.release.win32.x86.o):hostip.c:(.text+0x3e5): undefined reference to `_time32'
C:/BlitzMax/mod/bah.mod/libcurl.mod/libcurl.release.win32.x86.a(cookie.c.release.win32.x86.o):cookie.c:(.text+0x220): undefined reference to `_time32'
C:/BlitzMax/mod/bah.mod/libcurl.mod/libcurl.release.win32.x86.a(cookie.c.release.win32.x86.o):cookie.c:(.text+0x67c): more undefined references to `_time32' follow
C:/BlitzMax/mod/bah.mod/libcurl.mod/libcurl.release.win32.x86.a(parsedate.c.release.win32.x86.o):parsedate.c:(.text+0x86d): undefined reference to `_gmtime32'
C:/BlitzMax/mod/bah.mod/libcurl.mod/libcurl.release.win32.x86.a(ftp.c.release.win32.x86.o):ftp.c:(.text+0x48fc): undefined reference to `_time32'
C:/BlitzMax/mod/gman.mod/zipengine.mod/zipengine.release.win32.x86.a(bmxsupport.c.release.win32.x86.o):bmxsupport.c:(.text+0x49): undefined reference to `_mktime32'
C:/BlitzMax/mod/gman.mod/zipengine.mod/zipengine.release.win32.x86.a(zip.c.release.win32.x86.o):zip.c:(.text+0x228a): undefined reference to `_time32'
C:/BlitzMax/mod/bah.mod/fmod.mod/fmod.release.win32.x86.a(glue.cpp.release.win32.x86.o):glue.cpp:(.text+0x133): undefined reference to `operator delete(void*, unsigned int)'
C:/BlitzMax/mod/bah.mod/fmod.mod/fmod.release.win32.x86.a(glue.cpp.release.win32.x86.o):glue.cpp:(.text+0x18c): undefined reference to `operator delete(void*, unsigned int)'
C:/BlitzMax/mod/bah.mod/fmod.mod/fmod.release.win32.x86.a(glue.cpp.release.win32.x86.o):glue.cpp:(.text+0x273): undefined reference to `operator delete(void*, unsigned int)'
C:/BlitzMax/mod/bah.mod/fmod.mod/fmod.release.win32.x86.a(glue.cpp.release.win32.x86.o):glue.cpp:(.text+0x403): undefined reference to `operator delete(void*, unsigned int)'
C:/BlitzMax/mod/bah.mod/fmod.mod/fmod.release.win32.x86.a(glue.cpp.release.win32.x86.o):glue.cpp:(.text+0xe43): undefined reference to `operator delete(void*, unsigned int)'
C:/BlitzMax/mod/bah.mod/fmod.mod/fmod.release.win32.x86.a(glue.cpp.release.win32.x86.o):glue.cpp:(.text+0xeec): more undefined references to `operator delete(void*, unsigned int)' follow
C:/BlitzMax/mod/pub.mod/libpng.mod/libpng.release.win32.x86.a(pngwrite.c.release.win32.x86.o):pngwrite.c:(.text+0x9a0): undefined reference to `_gmtime32'
C:/BlitzMax/mod/pub.mod/stdc.mod/stdc.release.win32.x86.a(stdc.c.release.win32.x86.o):stdc.c:(.text+0xf61): undefined reference to `_time32'
C:/BlitzMax/mod/pub.mod/stdc.mod/stdc.release.win32.x86.a(stdc.c.release.win32.x86.o):stdc.c:(.text+0xf71): undefined reference to `_localtime32'
collect2.exe: error: ld returned 1 exit status

braxton

(Posted April) [#14]

Hi Grisu,

Sure, if you checkout the official git repo from here,
https://github.com/blitz-research/blitzmax

eg checkout to D:\BlitzMax

then open a command prompt where you just checked out and set your MinGW path to point to where you installed the i686-6.3.0-release-posix-dwarf-rt_v5-rev1

eg:
set MinGW=D:\i686-6.3.0-release-posix-dwarf-rt_v5-rev1\mingw32

then as per the included readme.txt for installing on windows

cd _src\win32_x86
install

and then everything should start building :)

I would definitely do a clean checkout and not on top of an already existing BlitzMax as it uses ar.exe and ld.exe and the libs from the MinGW installation similar to Brucey's, the only one it seems to copy into libs is libunicows

Grisu

(Posted April) [#15]

Worked fine. Only nitbit, the GCC version isn't showing up in the MAX IDE.

Also I used a much newer version of FASM: http://flatassembler.net/fasmw17160.zip

Thanks for the instructions. Might be helpful to a lot of other users too.

braxton

(Posted April) [#16]

Hi Grisu,

That is good news, I am glad to hear that you got it working, I also use the latest FASM and UPX.

If you add these to a 'launch_blitzmax_ide_630.bat' file then that should fix the version displayed in the original maxide.exe about dialog and allow you to have other versions installed all independently of each other, it is what I do when testing out new versions :)

eg:
set MinGW=D:\i686-6.3.0-release-posix-dwarf-rt_v5-rev1\mingw32
path %path%;%MinGW%\bin
maxide.exe

Grisu

(Posted April) [#17]

Can't get it to run properly: No GCC version number displayed.

But as long as the compiler itself works. I'm fine. :)

P.S. PRP gained about 200kb of weight from 5.1. to 6.3.

braxton

(Posted April) [#18]

Hi Brucey,

Thank you so very much it works really well using
x86_64-6.3.0-release-posix-seh-rt_v5-rev1 for the x64 MinGW version

I couldn't get any x86 versions to behave though, whichever one I tried when building threw the following message, possibly because all of them seem to still be using
'lib/gcc/i686-w64-mingw32'

Could not determine MinGWCrtPath: Expecting 'D:/BlitzMaxNG/MinGW32x86/lib/gcc/mingw32/6.3.0'

@Grisu I also noticed a small increase but the speed seems to have increased too, it appears to build and link a whole lot quicker.

Grisu

(Posted April) [#19]

@Braxton: I have noticed the same. Also the window resizing / window event handling of gui elements is now a lot smoother. No more "jumping around" buttons and such. Not sure if the latest Win10 build has caused this or the GCC compiler update though.

Grisu

(Posted May) [#20]

While all modules build fine, I get these errors when trying to compile the Guetzli example code:

Building example_01
Compiling:example_01.bmx
flat assembler  version 1.71.62  (1048576 kilobytes memory)
3 passes, 725 bytes.
Linking:example_01.exe
C:/BlitzMax/mod/bah.mod/guetzli.mod/guetzli.release.win32.x86.a(glue.cpp.release.win32.x86.o):glue.cpp:(.text+0xc0): undefined reference to `guetzli::ButteraugliScoreForQuality(double)'
C:/BlitzMax/mod/bah.mod/guetzli.mod/guetzli.release.win32.x86.a(glue.cpp.release.win32.x86.o):glue.cpp:(.text+0x159): undefined reference to `guetzli::Process(guetzli::Params const&, guetzli::ProcessStats*, std::vector<unsigned char, std::allocator<unsigned char> > const&, int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)'
collect2.exe: error: ld returned 1 exit status
Build Error: Failed to link C:/BlitzMax/mod/bah.mod/guetzli.mod/examples/example_01.exe
Process complete

Sorry to bring this thread up again. :( Perhaps there's a simple fix for it.

P.S.: I still use BMX vanilla with GCC 7.1 (32 Bit).