Bug 354585

Summary:	IPv6 connections to https://dot.kde.org fail
Product:	[Websites] www.kde.org	Reporter:	Jon Burgess <jburgess777>
Component:	general	Assignee:	kde-www mailing-list <kde-www>
Status:	RESOLVED FIXED
Severity:	normal	CC:	awilfox, bcooksley, elizabeth, kde, stupor_scurvy343
Priority:	NOR
Version:	unspecified
Target Milestone:	---
Platform:	Other
OS:	Linux
URL:	https://dot.kde.org
Latest Commit:		Version Fixed In:
Sentry Crash Report:

Description Jon Burgess 2015-10-29 23:55:22 UTC

I have been unable to connect successfully to https://dot.kde.org for several weeks. Digging a little deeper shows that the DNS entry returns both A and AAAA records and my machine tries to use the IPv6 (AAAA) result, forcing it to use IPv4 instead works.

Reproducible: Always

Steps to Reproduce:
1. Use firefox to connect to https://dot.kde.org from machine connected to internet via a dual stack IPv4 + IPv6


Actual Results:  
Connection times out

Expected Results:  
web page

The debug steps I went through are below. Although I can establish a TCP connection to port 443 OpenSSL fails to complete the HTTPS handshake with the server.

[jburgess@localhost ~]$ host dot.kde.org
dot.kde.org is an alias for t6jrh.x.incapdns.net.
t6jrh.x.incapdns.net has address 149.126.74.103
t6jrh.x.incapdns.net has IPv6 address 2a02:e980:c::67

[jburgess@localhost ~]$ ping dot.kde.org
PING t6jrh.x.incapdns.net (149.126.74.103) 56(84) bytes of data.
64 bytes from 149.126.74.103.ip.incapdns.net (149.126.74.103): icmp_seq=1 ttl=60 time=11.3 ms
64 bytes from 149.126.74.103.ip.incapdns.net (149.126.74.103): icmp_seq=2 ttl=60 time=10.8 ms
^C
--- t6jrh.x.incapdns.net ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 10.815/11.104/11.394/0.308 ms

[jburgess@localhost ~]$ ping6 dot.kde.org
PING dot.kde.org(2a02:e980:c::67) 56 data bytes
64 bytes from 2a02:e980:c::67: icmp_seq=1 ttl=60 time=11.1 ms
64 bytes from 2a02:e980:c::67: icmp_seq=2 ttl=60 time=10.7 ms
64 bytes from 2a02:e980:c::67: icmp_seq=3 ttl=60 time=11.0 ms
64 bytes from 2a02:e980:c::67: icmp_seq=4 ttl=60 time=11.0 ms
^C
--- dot.kde.org ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 10.721/10.973/11.152/0.202 ms

OpenSSL connecting IPv4 is OK:

[jburgess@localhost ~]$ openssl s_client -connect 149.126.74.103:443
CONNECTED(00000003)                                                                                                                                                  
depth=2 C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA                                                                                          
verify return:1                                                                                                                                                      
depth=1 C = BE, O = GlobalSign nv-sa, CN = GlobalSign Organization Validation CA - SHA256 - G2                                                                       
verify return:1                                                                                                                                                      
depth=0 C = US, ST = Delaware, L = Dover, O = Incapsula Inc, CN = incapsula.com                                                                                      
verify return:1                                                                                                                                                      
---                                                                                                                                                                  
Certificate chain                                                                                                                                                    
 0 s:/C=US/ST=Delaware/L=Dover/O=Incapsula Inc/CN=incapsula.com                                                                                                      
   i:/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Organization Validation CA - SHA256 - G2
 1 s:/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Organization Validation CA - SHA256 - G2
   i:/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
... content deleted ...
-----END CERTIFICATE-----
subject=/C=US/ST=Delaware/L=Dover/O=Incapsula Inc/CN=incapsula.com
issuer=/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Organization Validation CA - SHA256 - G2
---
No client certificate CA names sent
Server Temp Key: ECDH, prime256v1, 256 bits
---
SSL handshake has read 4910 bytes and written 333 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 27543796A9A45E09435BD0CEDD01932F78C291BC17D50CC46F04F0B0FF856F0F
    Session-ID-ctx: 
    Master-Key: 202005B53ED8D5D9067686AEF12386EE6F2D431D1D89A9FF97054679F253A78404DFFBFA5ACE4363112D44799FEB24C0
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - a1 dd 57 fb 6a 56 8b 21-8c 4d da 06 06 75 4f 3d   ..W.jV.!.M...uO=
    0010 - 38 1c d7 f1 b8 b4 77 19-35 81 c3 2e 7c e2 72 0c   8.....w.5...|.r.
    0020 - 15 a7 be ae 5a ff 51 9a-e6 0d 5a 65 b2 cb ff 58   ....Z.Q...Ze...X
    0030 - 2c e9 c7 cb 1c fb c0 eb-9b c5 1b af bd af 2e 8e   ,...............
    0040 - 25 9e 4c 08 9d 17 3c c8-ce 16 fb fb d3 70 f3 f0   %.L...<......p..
    0050 - 7a a4 ff 94 36 cf 22 cc-ae 19 65 8c 88 a1 00 5b   z...6."...e....[
    0060 - 01 57 b5 c3 19 75 72 03-49 13 a4 35 d9 de ba 4b   .W...ur.I..5...K
    0070 - 54 a5 35 b1 93 86 75 1e-9b 14 b4 70 2c b2 72 57   T.5...u....p,.rW
    0080 - 9e 09 d4 eb 2c 14 51 b9-d7 90 4c 82 14 0d 1a b6   ....,.Q...L.....
    0090 - a3 c9 27 d9 92 d6 0c 83-53 50 07 f1 19 5a 79 84   ..'.....SP...Zy.

    Start Time: 1446162215
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
DONE

Conecting via the IPv6 address times out...

[jburgess@localhost ~]$ openssl s_client -connect [2a02:e980:c::67]:443
CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 207 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

Similarly connecting via the DNS name (which is likely using IPv6) also times out:

[jburgess@localhost ~]$ openssl s_client -connect dot.kde.org:443
CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---                                                                                                                                                                  
SSL handshake has read 0 bytes and written 207 bytes                                                                                                                 
---                                                                                                                                                                  
New, (NONE), Cipher is (NONE)                                                                                                                                        
Secure Renegotiation IS NOT supported                                                                                                                                
Compression: NONE                                                                                                                                                    
Expansion: NONE                                                                                                                                                      
---

Comment 1 Jon Burgess 2015-10-30 00:04:33 UTC

wget http://dot.kde.org gets redirected to use https which then fails...

[jburgess@localhost ~]$ wget http://dot.kde.org
--2015-10-30 00:01:04--  http://dot.kde.org/
Resolving dot.kde.org (dot.kde.org)... 2a02:e980:c::67, 149.126.74.103
Connecting to dot.kde.org (dot.kde.org)|2a02:e980:c::67|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://dot.kde.org/ [following]
--2015-10-30 00:01:04--  https://dot.kde.org/
Connecting to dot.kde.org (dot.kde.org)|2a02:e980:c::67|:443... connected.
Unable to establish SSL connection.

Comment 2 Jon Burgess 2015-10-30 00:16:45 UTC

I see https:// working from another machine over IPv6. The machine that won't connect is on ADSL using PPPoE limited to 1492 byte frames. It could be a path MTU or ICMPv6 problem. I don't have any issues accessing other common IPv6 enabled sites like Google or facebook.

Comment 3 Jon Burgess 2015-10-30 00:56:29 UTC

I can confirm a path MTU issue is the cause of the problem. Sniffing working traffic to Google and Facebook show they both use an MSS clamp of 1410 bytes. If I add a local iptables rule to force a similar MSS on my traffic to dot.kde.org then it works as well:

$ sudo ip6tables -t mangle -A OUTPUT_direct -p tcp -d 2a02:e980:c::67 --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1410

$ wget -O /dev/null https://dot.kde.org
--2015-10-30 00:50:17--  https://dot.kde.org/
Resolving dot.kde.org (dot.kde.org)... 2a02:e980:c::67, 149.126.74.103
Connecting to dot.kde.org (dot.kde.org)|2a02:e980:c::67|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘/dev/null’

This suggests that there is probably a firewall blocking the ICMPv6 packets from getting back to your web server, breaking path MTU discovery. If you don't have control of this firewall then you might want to add a 1410 MSS clamp to your web server to workaround this problem for anyone else in a similar position to me.

Comment 4 Ben Cooksley 2015-10-30 07:30:06 UTC

Are you able to access https://krita.org without any clamping?

Comment 5 Ben Cooksley 2015-10-30 07:33:14 UTC

Also, cdn.kde.org doesn't redirect - can you try accessing http://cdn.kde.org/css/bootstrap.css and see if that loads without delay?

Comment 6 Jon Burgess 2015-10-31 15:22:28 UTC

Both the krita and cdn URLs load fine without clamping.

Comment 7 A. Wilcox (awilfox) 2015-12-10 21:03:13 UTC

For me, the Krita page loads fine, but the CDN server does not work under HTTP or HTTPS.  This has been going on since late October and is severely impacting my ability to file bugs, read the documentation about KF5 (porting my apps from kdelibs 4), and use KDE in general.  Is there anything I can do to help resolve this?  I don't have ip6tables in my kernel so unfortunately I can't test to see if that fixes it for me or not.

Comment 8 Ben Cooksley 2015-12-16 08:10:36 UTC

You could try artifically restraining your MTU, we're working with the provider on a fix though.

Comment 9 Elizabeth Myers 2016-03-16 19:29:03 UTC

Hi,

I've had problems too. No, setting MTU to the RFC minimum of 1280 didn't work. It shouldn't matter anyway unless the other end is configured incompetently.

I would say disable IPv6 for now until such a time this is actually resolved.

Comment 10 Ben Cooksley 2016-10-18 09:12:27 UTC

Hi all,

I've been informed by Incapsula that they've deployed changes which should resolve MTU and other similar issues relating to IPv6 accessibility.

Can you please test to confirm whether this is the case?

Comment 11 Chris 2016-10-18 11:04:30 UTC

Hi

I can confirm that I am now able to connect to https://dot.kde.org/ and https://cdn.kde.org/css/bootstrap.css via IPv6.

I had previously blocked 2a02:e980::/29 at my firewall to force IPv4 while this issue existed, but removing the block now allows IPv6 connectivity.

Thanks

Comment 12 Ben Cooksley 2016-10-19 09:26:23 UTC

Thanks Chris.

Based on another response I received via another channel which also indicated the access problems were fixed, combined with this one, i'm going to consider the issue resolved.