I set KMail to offline mode and confirmed that pop3 went to offline. Still, it keeps asking for the password from kwallet. I probably entered the password, and it downloaded the mails. :( This is very bad as (in my case) cause two computers to be out of sync regarding the downloaded mails.
Seems like bug 254395. Please re-test with a newer version and re-open this bug report if it still happens. *** This bug has been marked as a duplicate of bug 254395 ***