Bug 110980

Summary: kntlm broken on big-endian systems
Product: [Unmaintained] kio Reporter: Aaron Williams <aaronw>
Component: generalAssignee: Szombathelyi György <gyurco>
Status: RESOLVED FIXED    
Severity: normal    
Priority: NOR    
Version First Reported In: unspecified   
Target Milestone: ---   
Platform: unspecified   
OS: Solaris   
Latest Commit: Version Fixed In:
Sentry Crash Report:
Attachments: Patch to fix big endian support for kntlm.cpp
Patch that fixes Solaris and bug 93454
Patch that fixes Solaris and bug 93454

Description Aaron Williams 2005-08-18 02:31:50 UTC 
Version:           unknown (using KDE 3.4.2, compiled sources)
Compiler:          gcc version 3.3.6
OS:                SunOS (sun4u) release 5.8

kntlm does not work when running on Solaris running on Sparc.  Looking at a packet dump, I see that the length and offset fields are in big endian format when they should be in little endian format.  Firefox works, but kntlm uses the wrong byte order.

Looking at the code I have traced this to kdelibs/kio/misc/kntlm/kntlm.cpp   KNTLM::addBuf

I think I have a fix and will attach a patch soon to this bug.
Comment 1 Aaron Williams 2005-08-23 02:01:43 UTC 
I have made some changes to kntlm.cpp which appears to fix the endian issues, however, I am still having problems, but the problem is identical to what I am seeing on Linux X86 and is not endian specific (see bug 93454 which should be reopened).  I will attach my patch shortly.
Comment 2 Aaron Williams 2005-08-23 02:02:56 UTC 
Created attachment 12335 [details]
Patch to fix big endian support for kntlm.cpp

Here is my patch so far to fix the endian problems with kntlm.cpp.
Comment 3 Aaron Williams 2005-08-23 07:27:59 UTC 
Created attachment 12338 [details]
Patch that fixes Solaris and bug 93454

This patch got connectivity working from my big-endian Solaris box to our
Exchange server.  This also makes bug 93454 work but is a quick hack to just
work around NTLMv2 not working properly.  Note that NTLMv2 does not work on a
little endian system either (i586 SuSE Linux KDE 3.4.2)
Comment 4 Aaron Williams 2005-08-23 21:37:52 UTC 
Created attachment 12353 [details]
Patch that fixes Solaris and bug 93454

This patch got connectivity working from my big-endian Solaris box to our
Exchange server.  This also makes bug 93454 work but is a quick hack to just
work around NTLMv2 not working properly.  Note that NTLMv2 does not work on a
little endian system either (i586 SuSE Linux KDE 3.4.2)
Comment 5 Szombathelyi György 2005-08-27 21:21:42 UTC 
SVN commit 454038 by gyurco:

Fixes for big-endian archs. However, NTLMv2 problems needs a fix, too.

BUG: 110980


 M  +31 -17    kntlm.cpp  


--- branches/KDE/3.5/kdelibs/kio/misc/kntlm/kntlm.cpp #454037:454038
@@ -34,17 +34,21 @@
 
 QString KNTLM::getString( const QByteArray &buf, const SecBuf &secbuf, bool unicode )
 {
+  Q_UINT32 offset;
+  Q_UINT16 len;
+  offset = KFromToLittleEndian((Q_UINT32)secbuf.offset);
+  len = KFromToLittleEndian(secbuf.len);
   //watch for buffer overflows
-  if ( secbuf.offset > buf.size() ||
-       secbuf.offset + secbuf.len > buf.size() ) return QString::null;
+  if ( offset > buf.size() ||
+       offset + len > buf.size() ) return QString::null;
 
   QString str;
-  const char *c = buf.data() + secbuf.offset;
+  const char *c = buf.data() + offset;
   
   if ( unicode ) {
-    str = UnicodeLE2QString( (QChar*) c, secbuf.len >> 1 );
+    str = UnicodeLE2QString( (QChar*) c, len >> 1 );
   } else {
-    str = QString::fromLatin1( c, secbuf.len );
+    str = QString::fromLatin1( c, len );
   }
   return str;
 }
@@ -52,10 +56,14 @@
 QByteArray KNTLM::getBuf( const QByteArray &buf, const SecBuf &secbuf )
 {
   QByteArray ret;
+  Q_UINT32 offset;
+  Q_UINT16 len;
+  offset = KFromToLittleEndian((Q_UINT32)secbuf.offset);
+  len = KFromToLittleEndian(secbuf.len);
   //watch for buffer overflows
-  if ( secbuf.offset > buf.size() ||
-       secbuf.offset + secbuf.len > buf.size() ) return ret;
-  ret.duplicate( buf.data() + secbuf.offset, buf.size() );
+  if ( offset > buf.size() ||
+       offset + len > buf.size() ) return ret;
+  ret.duplicate( buf.data() + offset, buf.size() );
   return ret;
 }
 
@@ -77,11 +85,17 @@
 
 void KNTLM::addBuf( QByteArray &buf, SecBuf &secbuf, QByteArray &data )
 {
-  secbuf.offset = (buf.size() + 1) & 0xfffffffe;
-  secbuf.len = data.size();
-  secbuf.maxlen = data.size();
-  buf.resize( secbuf.offset + data.size() );
-  memcpy( buf.data() + secbuf.offset, data.data(), data.size() );
+  Q_UINT32 offset;
+  Q_UINT16 len, maxlen;
+  offset = (buf.size() + 1) & 0xfffffffe;
+  len = data.size();
+  maxlen = data.size();
+  
+  secbuf.offset = KFromToLittleEndian((Q_UINT32)offset);
+  secbuf.len = KFromToLittleEndian(len);
+  secbuf.maxlen = KFromToLittleEndian(maxlen);
+  buf.resize( offset + len );
+  memcpy( buf.data() + offset, data.data(), data.size() );
 }
 
 bool KNTLM::getNegotiate( QByteArray &negotiate, const QString &domain, const QString &workstation, Q_UINT32 flags )
@@ -118,7 +132,7 @@
   //challenge structure too small
   if ( chsize < 32 ) return false;
 
-  unicode = ch->flags & Negotiate_Unicode;
+  unicode = KFromToLittleEndian(ch->flags) & Negotiate_Unicode;
   if ( domain.isEmpty() )
     dom = getString( challenge, ch->targetName, unicode );
   else
@@ -130,8 +144,8 @@
   ((Auth*) rbuf.data())->flags = ch->flags;
   QByteArray targetInfo = getBuf( challenge, ch->targetInfo );
 
-  if ( forceNTLMv2 || (!targetInfo.isEmpty() && (ch->flags & Negotiate_Target_Info)) /* may support NTLMv2 */ ) {
-    if ( ch->flags & Negotiate_NTLM ) {
+  if ( forceNTLMv2 || (!targetInfo.isEmpty() && (KFromToLittleEndian(ch->flags) & Negotiate_Target_Info)) /* may support NTLMv2 */ ) {
+    if ( KFromToLittleEndian(ch->flags) & Negotiate_NTLM ) {
       if ( targetInfo.isEmpty() ) return false;
       response = getNTLMv2Response( dom, user, password, targetInfo, ch->challengeData );
       addBuf( rbuf, ((Auth*) rbuf.data())->ntResponse, response );
@@ -143,7 +157,7 @@
         return false;
     }
   } else { //if no targetinfo structure and NTLMv2 or LMv2 not forced, try the older methods
-    if ( ch->flags & Negotiate_NTLM ) {
+    if ( KFromToLittleEndian(ch->flags) & Negotiate_NTLM ) {
       response = getNTLMResponse( password, ch->challengeData );
       addBuf( rbuf, ((Auth*) rbuf.data())->ntResponse, response );
     } else {
Comment 6 Szombathelyi György 2005-08-27 21:24:02 UTC 
SVN commit 454042 by gyurco:

Backport fix for #110980 (endiannes problem)

CCBUG: 110980


 M  +31 -17    kntlm.cpp  


--- branches/KDE/3.4/kdelibs/kio/misc/kntlm/kntlm.cpp #454041:454042
@@ -35,16 +35,20 @@
 QString KNTLM::getString( const QByteArray &buf, const SecBuf &secbuf, bool unicode )
 {
   //watch for buffer overflows
-  if ( secbuf.offset > buf.size() ||
-       secbuf.offset + secbuf.len > buf.size() ) return QString::null;
+  Q_UINT32 offset;
+  Q_UINT16 len;
+  offset = KFromToLittleEndian((Q_UINT32)secbuf.offset);
+  len = KFromToLittleEndian(secbuf.len);
+  if ( offset > buf.size() ||
+       offset + len > buf.size() ) return QString::null;
 
   QString str;
-  const char *c = buf.data() + secbuf.offset;
+  const char *c = buf.data() + offset;
   
   if ( unicode ) {
-    str = UnicodeLE2QString( (QChar*) c, secbuf.len >> 1 );
+    str = UnicodeLE2QString( (QChar*) c, len >> 1 );
   } else {
-    str = QString::fromLatin1( c, secbuf.len );
+    str = QString::fromLatin1( c, len );
   }
   return str;
 }
@@ -52,10 +56,14 @@
 QByteArray KNTLM::getBuf( const QByteArray &buf, const SecBuf &secbuf )
 {
   QByteArray ret;
+  Q_UINT32 offset;
+  Q_UINT16 len;
+  offset = KFromToLittleEndian((Q_UINT32)secbuf.offset);
+  len = KFromToLittleEndian(secbuf.len);
   //watch for buffer overflows
-  if ( secbuf.offset > buf.size() ||
-       secbuf.offset + secbuf.len > buf.size() ) return ret;
-  ret.duplicate( buf.data() + secbuf.offset, buf.size() );
+  if ( offset > buf.size() ||
+       offset + len > buf.size() ) return ret;
+  ret.duplicate( buf.data() + offset, buf.size() );
   return ret;
 }
 
@@ -77,11 +85,17 @@
 
 void KNTLM::addBuf( QByteArray &buf, SecBuf &secbuf, QByteArray &data )
 {
-  secbuf.offset = (buf.size() + 1) & 0xfffffffe;
-  secbuf.len = data.size();
-  secbuf.maxlen = data.size();
-  buf.resize( secbuf.offset + data.size() );
-  memcpy( buf.data() + secbuf.offset, data.data(), data.size() );
+  Q_UINT32 offset;
+  Q_UINT16 len, maxlen;
+  offset = (buf.size() + 1) & 0xfffffffe;
+  len = data.size();
+  maxlen = data.size();
+  
+  secbuf.offset = KFromToLittleEndian((Q_UINT32)offset);
+  secbuf.len = KFromToLittleEndian(len);
+  secbuf.maxlen = KFromToLittleEndian(maxlen);
+  buf.resize( offset + len );
+  memcpy( buf.data() + offset, data.data(), data.size() );
 }
 
 bool KNTLM::getNegotiate( QByteArray &negotiate, const QString &domain, const QString &workstation, Q_UINT32 flags )
@@ -118,7 +132,7 @@
   //challenge structure too small
   if ( chsize < 32 ) return false;
 
-  unicode = ch->flags & Negotiate_Unicode;
+  unicode = KFromToLittleEndian(ch->flags) & Negotiate_Unicode;
   if ( domain.isEmpty() )
     dom = getString( challenge, ch->targetName, unicode );
   else
@@ -130,8 +144,8 @@
   ((Auth*) rbuf.data())->flags = ch->flags;
   QByteArray targetInfo = getBuf( challenge, ch->targetInfo );
 
-  if ( forceNTLMv2 || (!targetInfo.isEmpty() && (ch->flags & Negotiate_Target_Info)) /* may support NTLMv2 */ ) {
-    if ( ch->flags & Negotiate_NTLM ) {
+  if ( forceNTLMv2 || (!targetInfo.isEmpty() && (KFromToLittleEndian(ch->flags) & Negotiate_Target_Info)) /* may support NTLMv2 */ ) {
+    if ( KFromToLittleEndian(ch->flags) & Negotiate_NTLM ) {
       if ( targetInfo.isEmpty() ) return false;
       response = getNTLMv2Response( dom, user, password, targetInfo, ch->challengeData );
       addBuf( rbuf, ((Auth*) rbuf.data())->ntResponse, response );
@@ -143,7 +157,7 @@
         return false;
     }
   } else { //if no targetinfo structure and NTLMv2 or LMv2 not forced, try the older methods
-    if ( ch->flags & Negotiate_NTLM ) {
+    if ( KFromToLittleEndian(ch->flags) & Negotiate_NTLM ) {
       response = getNTLMResponse( password, ch->challengeData );
       addBuf( rbuf, ((Auth*) rbuf.data())->ntResponse, response );
     } else {