Bug 153973

Summary: Automatic detection and script based proxy config not working
Product: kio Reporter: Luiz Angelo De Luca <luizluca>
Component: generalAssignee: David Faure <faure>
Status: RESOLVED FIXED    
Severity: normal CC: adawit, tcarter, xeros
Priority: NOR    
Version: unspecified   
Target Milestone: ---   
Platform: unspecified   
OS: Linux   
Latest Commit: Version Fixed In: 4.7

Description Luiz Angelo De Luca 2007-12-13 16:23:01 UTC
Version:           3.97.00 (KDE 4.0 RC2) (using KDE 3.97.00 (KDE 4.0 RC2), compiled sources)
Compiler:          gcc
OS:                Linux (x86_64) release 2.6.22.13-0.3-default

Hello

   Konq on RC2 still does not accept "detect proxy config" and the script based config (informing the URL). It does not asks the HTTP host with the script config even after a login/loogut.
Comment 1 Fabio Correa 2008-01-24 18:09:08 UTC
Same happens at KDE-4.0.0 compiled from sources (ebuild) on a Gentoo box. x86_64-pc-linux-gnu-gcc-4.1.1
Comment 2 Tomasz Czapiewski 2010-02-05 13:40:20 UTC
I'm getting the same problem with KDE 4.3.5 and KDE 4.4rc2, too (on two Kubuntu 9.10 AMD64 machines).
I've configured DHCP and DNS for both PAC and WPAD methods.
All other browsers work fine with my configuration on both Linux and Windows.
Part of dhcpd.conf:
option domain-name "aaaaa.bbbb.ccc.dd";
option domain-name-servers 10.0.0.1;
option local-pac-server code 252 = text;        # automatic proxy settings
#option local-pac-server "http://10.0.0.1:8085/proxy/proxy.pac"; # not for firefox
#option local-pac-server "http://10.0.0.1:8085/wpad.dat";
#option local-pac-server "http://10.0.0.1:8085/proxy.pac";
#option local-pac-server "http://10.0.0.1:80/proxy.pac";
#option local-pac-server "http://10.0.0.1/proxyconfig.pac";
option local-pac-server "http://10.0.0.1:80/proxyconfig.pac";
# WPAD definition
option wpad code 252 = text;
# Suppress WPAD activity - no cache, no DNS.
#option wpad "\n\000";
# Configure a valid WPAD cache.  The \n is required for Windows.
# All config below this line is optional.
#option wpad "http://10.0.0.13:8085/wpad.dat\n";
#option wpad "http://10.0.0.13/wpad.dat\n";
#option wpad "http://10.0.0.13:80/wpad.dat";
#option wpad "http://10.0.0.13/proxyconfig.pac";
option wpad "http://10.0.0.13:80/proxyconfig.pac";

I've tested all the commented settings.
PCs are using DHCP+DNS for IP and domain names, apache serves all the files above on both ports: 8085 and 80.
I get the files (on PCs) by:
wget -S --spider http://wpad/wpad.dat
wget -S --spider http://10.0.0.1/wpad.dat
wget -S --spider http://wpad.aaaaa.bbbb.ccc.dd/wpad.dat
the same with proxyconfig.pac and proxy.pac

$ LC_ALL=C wget -S --spider http://wpad/wpad.dat
Spider mode enabled. Check if remote file exists.
--2010-02-05 13:26:31--  http://wpad/wpad.dat
Resolving wpad... 10.0.0.1
Connecting to wpad|10.0.0.1|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Date: Fri, 05 Feb 2010 12:26:31 GMT
  Server: Apache/2.2.12 (Ubuntu)
  Last-Modified: Fri, 05 Feb 2010 09:37:38 GMT
  ETag: "55819b-46-47ed733bdfc80"
  Accept-Ranges: bytes
  Content-Length: 70
  Keep-Alive: timeout=15, max=100
  Connection: Keep-Alive
  Content-Type: application/x-ns-proxy-autoconfig
Length: 70 [application/x-ns-proxy-autoconfig]
Remote file exists.

$ LC_ALL=C wget -S --spider http://wpad/proxy.pac
Spider mode enabled. Check if remote file exists.
--2010-02-05 13:27:26--  http://wpad/proxy.pac
Resolving wpad... 10.0.0.1
Connecting to wpad|10.0.0.1|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Date: Fri, 05 Feb 2010 12:27:26 GMT
  Server: Apache/2.2.12 (Ubuntu)
  Last-Modified: Fri, 05 Feb 2010 09:37:38 GMT
  ETag: "55819b-46-47ed733bdfc80"
  Accept-Ranges: bytes
  Content-Length: 70
  Keep-Alive: timeout=15, max=100
  Connection: Keep-Alive
  Content-Type: application/x-ns-proxy-autoconfig
Length: 70 [application/x-ns-proxy-autoconfig]
Remote file exists.

When I check traffic on server with iptraf I see always DNS connection from Konqueror to my DNS server UDP to port 53 and (only sometimes traffic UDP 67-68 ports - DHCP).
I've checked that /usr/bin/kpac_dhcp_helper is suid root and can be executed by user.
After each try I kill konqueror, kdeinit4: kio_file, kio_http processes and sometimes restart PC.
I've dropped suid from kpac_dhcp_helper to check if it can handle WPAD by DNS and not DHCP but it's the same as with DHCP - even the traffic looks the same - no connection to apache for autoproxy config.

My .pac and .dat files contain only:
function FindProxyForURL(url, host)
{
return "PROXY 10.0.0.1:3128";
}

And I've checked 'return "PROXY 10.0.0.1:3128; DIRECT";' or 'return "PROXY 10.0.0.1:3128"; return "DIRECT";', too before that I've realized that Konqueror doesn't even check for that file.

I've got no problems with that configuration on Chromium/Chrome 4.x/5.x (both Windows XP and Kubuntu), Firefox 3.6 (Win&Lin) and IE8 (WinXP).
Comment 3 Tomasz Czapiewski 2010-02-05 13:44:22 UTC
I've missed IP in config (my server IP was 10.0.0.13 ;-) not 10.0.0.1 - I've tried to change it by hand in config before putting it here and I've missed some entries so that's why there are different IPs in the config here but it's OK (the same IP) on my server.
But the problem is real - it was not a bug in my server's configuration.
Comment 4 Tomasz Czapiewski 2010-02-05 13:54:05 UTC
$ konqueror                                                                                
konqueror(4219) KonqMisc::createBrowserWindowFromProfile: path= "/home/darek/.kde/share/apps/konqueror/profiles/webbrowsing" , filename= "webbrowsing" , url= KUrl("")                                                                                                                    
konqueror(4219) KonqFactory::createView: Trying to create view for "KonqAboutPage" "konq_aboutpage"                                          
konqueror(4219) KonqFactory::createView: Found requested service "konq_aboutpage"                                                            
konqueror(4219) KonqFactory::createView: Trying to open lib for requested service  "konq_aboutpage"                                          
konqueror(4219) KonqView::openUrl: url= KUrl("about:konqueror") locationBarURL= ""                                                           
konqueror(4219) kdemain: main() -> no args                                                                                                   
konqueror(4219) KonqMainWindow::openFilteredUrl: url "www.wp.pl" filtered into KUrl("http://www.wp.pl")                                      
konqueror(4219) KonqMainWindow::openUrl: Creating new konqrun for KUrl("http://www.wp.pl") req.typedUrl= "www.wp.pl"                         
konqueror(4219) KonqRun::slotRedirection: KUrl("http://www.wp.pl") -> KUrl("http://www.wp.pl/")                                              
konqueror(4219) KonqFactory::createView: Trying to create view for "text/html" ""                                                            
konqueror(4219) KonqFactory::createView: "khtml"  : X-KDE-BrowserView-AllowAsDefault is valid :  false                                       
konqueror(4219) KonqView::openUrl: url= KUrl("error:/?error=123&errText=www.wp.pl: Connection refused#http://www.wp.pl") locationBarURL= "error:/?error=123&errText=www.wp.pl: Connection refused#http://www.wp.pl"                                                                       

To get logs I've checked konqueror, kio_http and kio_http_debug in kdebugdialog.
Comment 5 Tomasz Czapiewski 2010-02-05 14:08:48 UTC
And I'm getting nofication that "Konqueror could not find proper proxy autoconfiguration script" immediately after it tries to connect. It doesn't even try to search for it. It looks the same with both DHCP and DNS - not even a second wait to search for script. It's connection refused because I've dropped http traffic for these computers in firewall on gateway (different host than dhcp/dns/web/proxy server).
With putting static proxy settings in Konquerors configuration or even link to autoconfig script Konqueror opens web pages properly but I need to get proxy autoconfig from DHCP or DNS.
Comment 6 Travers Carter 2010-05-31 08:57:03 UTC
I have been investigating the same problem (proxy discovery not working) on Fedora 13 and found several problems affecting the proxy discovery process

1. kpac_dhcp_helper was not set-uid root, obviously this was a fedora packaging bug, but it could also be overlooked on other systems and it fails silently, breaking DHCP based discovery.

See https://bugzilla.redhat.com/show_bug.cgi?id=592658


2. A bug in Qt 4 means that it fails to parse the wpad URLs due to the trailing dot in the constructed WPAD URL (eg http://wpad.example.com./wpad.dat), this breaks DNS based discovery.

See http://bugreports.qt.nokia.com/browse/QTBUG-10972


3. kpac_dhcp_helper does a gethostname()+gethostbyname() to find the local IP to put into the DHCPINFORM query, but because this isn't necessarily reliable it can break DHCP discovery.

I have filed a NetworkManager but relating to this, but the maintainer suggested that even when fixed there are lots of scenarios where the gethostname()+gethostbyname() will still be wrong, eg when the hostname is localhost.localdomain.

See the Fedora bug above & http://bugreports.qt.nokia.com/browse/QTBUG-10972
Comment 7 Tomasz Czapiewski 2010-05-31 10:15:11 UTC
Thanks Travers for looking at the problem.
In my case it's the second bug I see. I've compiled yours code to try and its output on Kubuntu 10.04 (KDE 4.4.3 and QT 4.6.2) is the same as yours.
I'm not sure about last one... 

I'm not suer about the first one either, I had:
$ for i in `locate kpac_dhcp_helper`; do ls -al $i; done
-rwsr-xr-x 1 root root 10536 2010-01-29 02:14 /usr/bin/kpac_dhcp_helper
-rwxr-xr-x 1 root root 10528 2010-05-05 13:17 /usr/lib/kde4/libexec/kpac_dhcp_helper
-rw-r--r-- 1 root root 694 2010-01-29 02:13 /usr/share/man/man1/kpac_dhcp_helper.1.gz

Now I've set-uid root the second kpac_dhcp_helper in /usr/lib/kde4/libexec/kpac_dhcp_helper to test.
Comment 8 Tomasz Czapiewski 2010-05-31 10:37:32 UTC
Setting set-uid root bit on /usr/lib/kde4/libexec/kpac_dhcp_helper didn't help either. 

Btw. how this problem could be still unconfirmed while it's confirmed by few people. Can we set it to confirmed somewhere in this bug report?
Comment 9 Tomasz Czapiewski 2010-07-29 08:02:40 UTC
It still doesn't work on KDE 4.4.92 (4.5RC2) with QT 4.7.0
Comment 10 Tomasz Czapiewski 2010-07-29 12:03:33 UTC
QT bug is fixed but even that it still doesn't work in KDE.
Comment 11 Dawit Alemayehu 2011-03-30 08:16:26 UTC
Can anyone that compiled from source, please the patch proposed at

https://git.reviewboard.kde.org/r/100964/

and report the result here ? I am planning a major overhaul of KIO's Proxy handling for KDE 4.7, but need feedback from people that use this on a daily basis.
Comment 12 Dawit Alemayehu 2011-04-07 07:15:07 UTC
Git commit 98946bcb75615b0900a2b2e124cae89b7c323fd0 by Dawit Alemayehu.
Committed on 29/03/2011 at 00:12.
Pushed by adawit into branch 'master'.

KIO proxy overhaul Part I:

Features:
- Switched the PAC file parser from KJS to QtScript.
- Implemented the Microsoft IPv6 PAC extensions.
- Implemented support for returning SOCKS proxy information (only SOCKS5).
- Added a new dbus call, proxiesForUrl, that returns all the proxy addresses
  for a given url.
- Added a new kpactest file to test the new MS IPv6 extension functions.

Fixes:
- Fixed the DateRange function by adding back two missing months in the static
  months array.
- Fixed the DNS related functions by fixing how the lookup works through
  the following modification to the KIO DNS caching code:
    * Added a function, lookupCachedHostInfoFor, for quering the KIO cache for
      information without causing a DNS query.
    * Added a function, cacheLookup, for inserting a DNS lookup result into
      the KIO cache.
- Fixed myIpAddress by using QNetworkInterface, though this fix is still far
  from ideal since it grabs the first suitable interface on multi-homed systems.

REVIEW: 100971
BUG: 153973
BUG: 267844
FIXED-IN: 4.7
DIGEST: KIO Proxy overhaul for KDE 4.7

M  +32   -0    kio/kio/hostinfo.cpp     
M  +4    -0    kio/kio/hostinfo_p.h     
M  +2    -5    kio/misc/kpac/CMakeLists.txt     
M  +1    -0    kio/misc/kpac/kpac_dhcp_helper.c     
M  +2    -2    kio/misc/kpac/kpactest.pac     
A  +63   -0    kio/misc/kpac/kpactest2.pac         [License: UNKNOWN]  *
M  +138  -64   kio/misc/kpac/proxyscout.cpp     
M  +5    -2    kio/misc/kpac/proxyscout.h     
M  +640  -376  kio/misc/kpac/script.cpp     
M  +3    -3    kio/misc/kpac/script.h     

The files marked with a * at the end have a non valid license. Please read: http://techbase.kde.org/Policies/Licensing_Policy and use the headers which are listed at that page.


http://commits.kde.org/kdelibs/98946bcb75615b0900a2b2e124cae89b7c323fd0