Version: (using KDE KDE 3.4.2) Installed from: SuSE RPMs OS: Linux HTTPFilterGZip uses inflateInit2(&zstr, -MAX_WBITS) to initialize zlib. Even when -MAX_WBITS is used in some programs of zlib-source, it is not documented. Documentation tells, the value should be in the range 8..15. The value of -MAX_WBITS results in a Z_DATA_ERROR (-3), in inflate, when decompression is done. Using inflateInit(&zstr) sets the windowBit-parameter to the default value of MAX_WBITS, which should work. I have currently a server running at "http://hal.circle-of-art.de:8000/hello", which deliveres deflated data. This works fine with Firefox, but not with Konqueror (or other users of kio_http).
What zlib version do you have? (You can try to find the file /usr/include/zlib.h ) As far as I understand, this is a work-around for a bug in zlib. (No idea which version however.) So perhaps we could try to call inflateInit2(&zstr, -MAX_WBITS) and if it returns Z_DATA_ERROR, we could try to call it with MAX_WBITS. Have a nice day!
On Tuesday 06 December 2005 11:29, Nicolas Goutte wrote: (...) > ------- What zlib version do you have? (You can try to find the file > /usr/include/zlib.h ) I have downloaded the lastest zlib source (zlib 1.2.3) to look if there would be a difference to the zlib on my system. At first glance, there is not anything relevant for the problem. > > As far as I understand, this is a work-around for a bug in zlib. (No idea > which version however.) Here is the only changes in zlib 1.2.3. It documents correctly hwat the - sign means with this function. In fact, it is not a "work-around for a bug", it just tells the library not to expect a zlib header. > > So perhaps we could try to call inflateInit2(&zstr, -MAX_WBITS) and if it > returns Z_DATA_ERROR, we could try to call it with MAX_WBITS. So probably this is not needed. > However all this seems to be a bug report for an expert in HTTP, so I do not think that I cannot contribute much anymore. > Have a nice day!
From what I see the problem is that the handling of deflate is done by the class handling gzip. So the zlib is initialized without a zlib header, which is needed for gzip but which of course is bad for HTTP deflate.
Created attachment 13786 [details] Hack to fix HTTP deflate The patch makes the site works but Konqy complains by an unexpected end of file.
Created attachment 13787 [details] Another patch I have now another patch. As I am not sure to go into the right direction, I will better stop here.
SVN commit 486068 by goutte: HACK: I have realized that there was a brute force way of fixing bug #117683 at least temporarily: not to accept "deflate" CCBUG:117683 M +3 -1 http.cc --- branches/KDE/3.5/kdelibs/kioslave/http/http.cc #486067:486068 @@ -2393,7 +2393,9 @@ #ifdef DO_GZIP if (m_request.allowCompressedPage) - header += "Accept-Encoding: x-gzip, x-deflate, gzip, deflate\r\n"; + header += "Accept-Encoding: x-gzip, gzip\r\n"; + // ### FIXME: HTTP deflate support does not work, see KDE bug #117683 + // header += "Accept-Encoding: x-gzip, x-deflate, gzip, deflate\r\n"; #endif if (!m_request.charsets.isEmpty())
SVN commit 486096 by mueller: newer versions of zlib are a bit less buggy than the older ones, so disable some of our workaround code then. CCBUG: 117683 M +14 -0 kio/httpfilter/httpfilter.cc M +1 -3 kioslave/http/http.cc --- branches/KDE/3.5/kdelibs/kio/httpfilter/httpfilter.cc #486095:486096 @@ -105,7 +105,14 @@ zstr.zalloc = Z_NULL; zstr.zfree = Z_NULL; zstr.opaque = Z_NULL; + +#if ZLIB_VERNUM - 0 >= 0x1210 + /* zlib ver. >= 1.2.1 supports transparent gzip decompressing */ + inflateInit2(&zstr, MAX_WBITS+32); +#else inflateInit2(&zstr, -MAX_WBITS); +#endif + iTrailer = 8; #endif } @@ -210,6 +217,12 @@ return 2; } } + +#if ZLIB_VERNUM - 0 >= 0x1210 + /* zlib ver. >= 1.2.1 supports transparent gzip decompressing */ + return 0; +#else + int method = get_byte(); /* method byte */ int flags = get_byte(); /* flags byte */ @@ -237,6 +250,7 @@ } return bEof ? 2 : 0; +#endif #else return 0; #endif --- branches/KDE/3.5/kdelibs/kioslave/http/http.cc #486095:486096 @@ -2393,9 +2393,7 @@ #ifdef DO_GZIP if (m_request.allowCompressedPage) - header += "Accept-Encoding: x-gzip, gzip\r\n"; - // ### FIXME: HTTP deflate support does not work, see KDE bug #117683 - // header += "Accept-Encoding: x-gzip, x-deflate, gzip, deflate\r\n"; + header += "Accept-Encoding: x-gzip, x-deflate, gzip, deflate\r\n"; #endif if (!m_request.charsets.isEmpty())
On Tuesday 06 December 2005 20:33, Dirk Mueller wrote: [bugs.kde.org quoted mail] You have only changed something about gzip compressing not about deflate. > > if (!m_request.charsets.isEmpty()) Have a nice day!
On Tuesday 06 December 2005 19:50, Dirk Mueller wrote: > On Tuesday 06 December 2005 19:12, Nicolas Goutte wrote: > > HACK: I have realized that there was a brute force way of fixing bug > > #117683 at least temporarily: not to accept "deflate" > > ARGH. why do you do that? the server is broken. As I have seen in the meanwhile that the revert has been reverted, I suppose that I need to add something here. Yes, perhaps the server refered in the bug report is not 100% correct, as the zlib stream seems not to be closed. That is why I have stopped to try to fix the right code until somebody could correctly look at it. (By the way, if somebody knows a correct server that serves a correct "deflate" stream, I am interested in the URL to try to continue to fix the problem.) A few words to your revert. You have supposely added a feature of newer zlib libraries. However that was perhaps an improvement of the "gzip" streams (apparently not, as it was reverted again), but it does not help "deflate" streams in any way. I assume that you could read RFC 2616 as well as I have read it. (It is in kdelibs/kiobase/http.) You can read the section about "deflate". It refers to RFC 1950, the zlib format. However KDE uses RFC 1951. So KDE's implementation is broken! (Of course, some servers might serve RFC 1951 (named "deflate"), mixing up the "deflate" names, but that would be another of buggy server supports.) The differences are: - RFC 1951 is bare (supported by the - trick by zlib) - RFC 1950 has the zlib header (zlib default) - gzip has a gzip header (only newer zlib versions have full support for it) I would have fixed the "deflate" code if I would have found another server serving correct "deflate" streams to compare. I have not found one, so I prefered to stop. While reading RFC 2616 again, I have had the idea that the Accept-Encoding header told that KDE declares supporting "deflate". As it is not the case, it is easy and not intrusive to change it so that KDE does not declare supporting "deflate" anymore. I have never told that I had fixed the bug nor that I had broken gzip support (which is unaffected, as well by the bug than by my change). And the bug report is not closed either. If somebody can find a solution before KDE 3.5.1, very fine, otherwise the simple work-around will be in KDE 3.5.1. (It much better than KDE works with a server in uncompressed mode than to trigger an error that is unnecessary.) Have a nice day!
I implemented gzip-encoding in tntnet. It works now with kio_http. But only, because tntnet prefers gzip. The deflate-encoding was wrong too. I fixed it also, but kio_http still don't work. I don't know, what to do with deflate. I read rfc1950. The implementation of deflate in tntnet seems to be not correct. But when I follow rfc1950 firefox does not work with deflate (nor kio_http). I haven't found a way to get deflate with kio_http (unpatched) running. Deflate seems to be broken by default. Maybe we should forget about deflate. The first brute-force-solution (removing deflate from Accept-Encoding) seems to be best, as long as we don't know how to handle it right. I found a apache-module mod_deflate, which sounds good first. But it uses gzip-encoding - not deflate. Tommi
On Thursday 08 December 2005 17:13, Tommi "MÀkitalo" wrote: [bugs.kde.org quoted mail] Yes, many servers support "gzip". That is also one reason why I have chosen the way to deactivate "deflate", as it seems to have only a very limited impact. (Also kio_http does not support "compress" either, so it is not the first compression method that we skip.) > > I don't know, what to do with deflate. I read rfc1950. The implementation > of deflate in tntnet seems to be not correct. But when I follow rfc1950 > firefox does not work with deflate (nor kio_http). > I haven't found a way to > get deflate with kio_http (unpatched) running. The unpatched kio_http wants a "deflate" stream without any header (probably pure RFC 1951). > Deflate seems to be broken > by default. Maybe we should forget about deflate. The first > brute-force-solution (removing deflate from Accept-Encoding) seems to be > best, as long as we don't know how to handle it right. > > I found a apache-module mod_deflate, which sounds good first. But it uses > gzip-encoding - not deflate. That confirms my tests about finding a server serving "deflate". Apache will not serve it. > By the way, for testing I have used wget, with: wget --debug --header="Accept-Encoding: deflate" URL (URL being the URL to a site) Then you can easy see if the file is uncompressed or compressed. (Also --debug gives the header sent to server and the header received from the server.) And you can easily dump the compressed file if you had a change to get one. (wget by itself ha no support for compressed streams.) > Tommi Have a nice day!
Deflate is an interesting issue the Zlib faq states that mozilla is one of the few browsers to correctly implement deflate, and also that IE does not implement it correctly... for this reason most servers avoid it. >Yes, perhaps the server refered in the bug report is not 100% correct, as the >zlib stream seems not to be closed. for GZIP data Konqueror is complaining about incomplete input but I have specified the Content-Length: and served that number of bytes of compressed data. and it has correctly decoded them. what am I doing wrong? AHH! found it! I was ending the stream with deflate(&myzstrm,Z_FULL_FLUSH); // Don't do this. I changed to this and now it works. deflate(&myzstrm,Z_FINISH); // *** DO THIS *** mozilla and IE are less fussy... I've not checked the RFC's details. // I am using this for Gzip compression deflateInit2(&myzstrm,9,Z_DEFLATED,31,8,Z_DEFAULT_STRATEGY); // and this for deflate (only works with mozilla AFAICT) deflateInit2(&myzstrm,9,Z_DEFLATED,15,8,Z_DEFAULT_STRATEGY); // 9 is compression level, // 15/31 is method zlib-deflate or gzip, // -15 would give raw deflate but RFC-1952 calls for zlib deflate. // 8 is memory usage etc... see zlib.h for details. Konqueror works with raw deflate, so does mozilla (but mozilla also handles zlib deflate) Zlib deflate is apparently what the RFC Calls for, but given the number of broken implementations accepting either encoding would probably be a good idea. ==> http://www.zlib.net/zlib_faq.html#faq38
*** Bug 114830 has been marked as a duplicate of this bug. ***
*** Bug 188935 has been marked as a duplicate of this bug. ***
I'm working on a patch that makes deflate work, but I would welcome more testcases. http://hal.circle-of-art.de:8000/hello is not online anymore, does anyone have other testcases? Right now I only have the one from bug 188935 (serv-u.com).
Jasen Betts, thanks for a very complete analysis of the issue. That faq entry solved the whole mystery: Microsoft messed up again. And the only way to fix this correctly is to do some code to do auto-detection of zlib headers to see if we should initialize zlib with "there will be zlib headers" or "nope, just raw deflate". zlib doesn't seem to do that by itself. Did that, after porting the code to a higher-level C++ class around zlib I wrote some time ago. All my testcases work now.
SVN commit 951594 by dfaure: Port HTTPFilter to KGzipFilter (was using direct zlib calls previously, with lots of hacks), and add support for Microsoft's inability to read RFC 2616, i.e. detect raw-deflated data where deflated-with-zlib-headers data is normally expected. (This commit depends on r951584) BUG: 117683 M +1 -2 kdecore/compression/kfilterdev.h M +11 -7 kdecore/compression/kgzipfilter.cpp M +20 -3 kdecore/compression/kgzipfilter.h M +39 -0 kdecore/tests/kfiltertest.cpp M +1 -0 kdecore/tests/kfiltertest.h M +8 -0 kio/httpfilter/README M +104 -259 kio/httpfilter/httpfilter.cc M +20 -30 kio/httpfilter/httpfilter.h WebSVN link: http://websvn.kde.org/?view=rev&revision=951594