Can you tell us what kind of resource is the destination (where you download the mails, probably called Local Folders or KMail Folders). Start akonadiconsole, Agents tab, click on the account name and see what is the Type of it in the below pane.

I am seeing the same behavior on a newly built system. My home directory was copied over from my old system, but I deleted .kde4 & .local to get a clean start for KDE.

I have checked akonadiconsole and my email is going into "Local Folder" of type "maildir".

Further to the "mailing list" suggestion from the original reporter, I am also seeing the duplication on locally sourced messages where the "Subject:" header has text enclosed by square brackets, [like this].

In my case, doing almost anything to the inbox (such as deleting a message, or even selecting a different message) triggers the re-appearance (and re-processing by the filters) of over 300 messages.

On a related note. While trying to clean up after this I have noticed that the "remove duplicate messages" option from the folder menu doesn't seem to work on folders over a certain size. I have not done enough testing to determine where the cutoff point is.

It is working fine on some folders with under 100 messages, but doing nothing on other folders with over 3000 messages. It fails silently, no alerts from kmail, and no log entries I can find.

The local folder is a Maildir type, with some sub-folders, which receive the
messages (the inbox folder is empty) I too noted that only the messages with
square brackets are affected.
I also noted that the bug can be triggered with an explict update of the
folder, for example:

LocalFolder
   inbox
   ...
   Folder A
       Folder B

The messages go directly to Folder A and Folder B (via a filter). If I update
the Folder A the bug is triggered, if I update the Folder B the bugs is not
triggered

Disabling the automatic update of the Folder A and is a workaround, but even
after some days, if I update the Folder A, the old messages will reappear.

I tried to reproduce the same problem with a separate user but with no luck,
which led me to a difference between the two setups: my normal user use the
external mysql server (server version 5.5.19, archlinux package) , while the
test user use the internal mysql server.

Multiple copies of e-mails being downloaded is not a new problem. I have suffered this for many years with the old kmail. The problem would be most noticeable when kmail had not been run for a few days, when the catching-up process would produce up to half-a-dozen copies of each e-mail.

The only change I've noticed with the move to kmail2 is that I now get pop-up warnings of conflicting versions and am asked if I want to keep the left-hand one, or the right, or both - as if I know!

I keep my mail on my server for a while and don't delete it when it has been downloaded or deleted from my machine. This is because I'm almost always looking for an acceptable mail-and-news-reader and need to keep the ones I'm assessing in sync with each other.

I can confirm this and it's really severe.

KMail2 seems to ignore pop3 protocol completely. It downloads all mails from mailman mailing lists over and over again to the inbox.
It even downloads SPAM which I marked as deleted on the server.

Kmail from kde 3.5.10 does not.

Example of email headers for an email sent via mailman:

--start
Return-path: <phpugs-bounces@phpugs.de>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on www.points.de
X-Spam-Status: No, score=-2.3 required=3.0 tests=BAYES_00,HTML_MESSAGE
	autolearn=ham version=3.2.5
X-Spam-Level: 
Envelope-to: jo@feuersee.de
Delivery-date: Tue, 20 Dec 2011 20:12:05 +0100
Received: from shoptransfer.de ([88.198.23.111])
	by mail.points.de with esmtp (Exim 4.77)
	(envelope-from <phpugs-bounces@phpugs.de>)
	id 1Rd56b-0001Ve-Fk
	for jo@feuersee.de; Tue, 20 Dec 2011 20:12:05 +0100
Received: from localhost ([127.0.0.1] helo=[88.198.23.111])
	by shoptransfer.de with esmtp (Exim 4.72)
	(envelope-from <phpugs-bounces@phpugs.de>)
	id 1Rd56H-00010i-8s; Tue, 20 Dec 2011 20:11:49 +0100
Received: from mailout-de.gmx.net ([213.165.64.23])
	by shoptransfer.de with smtp (Exim 4.72)
	(envelope-from <jspranz@gmx.de>) id 1Rd34M-0008I7-IY
	for phpugs@phpugs.de; Tue, 20 Dec 2011 18:01:41 +0100
Received: (qmail invoked by alias); 20 Dec 2011 17:01:33 -0000
Received: from mail-iy0-f182.google.com (EHLO mail-iy0-f182.google.com)
	[209.85.210.182]
	by mail.gmx.net (mp069) with SMTP; 20 Dec 2011 18:01:33 +0100
X-Authenticated: #1046002
X-Provags-ID: V01U2FsdGVkX18I8CIi8IsIPC4kAz5mifWcWefEOHBCJjSeozwvTy
	zlxp/+gYMRQgMj
Received: by iadj38 with SMTP id j38so9555050iad.27
	for <phpugs@phpugs.de>; Tue, 20 Dec 2011 09:01:31 -0800 (PST)
MIME-Version: 1.0
Received: by 10.42.147.6 with SMTP id l6mr2426393icv.17.1324400491887; Tue, 20
	Dec 2011 09:01:31 -0800 (PST)
Received: by 10.50.189.201 with HTTP; Tue, 20 Dec 2011 09:01:31 -0800 (PST)
Date: Tue, 20 Dec 2011 18:01:31 +0100
Message-ID: <CAOYyNPrCDDDG8rOhAcvsPT6-5tgz3ObQqQFM_eh2m8ZC43Tdng@mail.gmail.com>
From: =?ISO-8859-1?Q?J=FCrgen_Spranz?= <jspranz@gmx.de>
To: phpugs@phpugs.de
X-Y-GMX-Trusted: 0
X-Virus-Scanned: Yes, by ClamAV
 
 Content preview:  content content 
content content content content content content content content content content content content content content content content 
newline
content content content content content content content content content content content content  [...] 
 
 Content analysis details:   (-1.9 points, 5.0 required)
 
  pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -0.0 RCVD_IN_DNSWL_NONE     RBL: Sender listed at http://www.dnswl.org/, low
                             trust
                             [213.165.64.23 listed in list.dnswl.org]
  0.0 FREEMAIL_FROM          Sender email is freemail (jspranz[at]gmx.de)
 -0.0 SPF_PASS               SPF: sender matches SPF record
  0.0 HTML_MESSAGE           BODY: HTML included in message
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
                             [score: 0.0000]
X-Mailman-Approved-At: Tue, 20 Dec 2011 20:11:33 +0100
Subject: [PHPUGS] Frameworks (Java Script)
X-BeenThere: phpugs@phpugs.de
X-Mailman-Version: 2.1.13
Precedence: list
Reply-To: Mailingliste der PHP User Group Stuttgart <phpugs@phpugs.de>
List-Id: Mailingliste der PHP User Group Stuttgart <phpugs.phpugs.de>
List-Unsubscribe: <https://ssl-factory.de/mailman/options/phpugs>,
	<mailto:phpugs-request@phpugs.de?subject=unsubscribe>
List-Archive: <https://ssl-factory.de/mailman/private/phpugs>
List-Post: <mailto:phpugs@phpugs.de>
List-Help: <mailto:phpugs-request@phpugs.de?subject=help>
List-Subscribe: <https://ssl-factory.de/mailman/listinfo/phpugs>,
	<mailto:phpugs-request@phpugs.de?subject=subscribe>
Content-Type: multipart/mixed; boundary="===============5207375469241522320=="
Sender: phpugs-bounces@phpugs.de
Errors-To: phpugs-bounces@phpugs.de
X-Virus-Scanned: Yes, by ClamAV
 
 Content preview:  Hallo, welches Framework würdet ihr einem "Neueinsteiger"
   empfehlen. Ich möchte einfach nicht alles bis ins Detail mit CSS, HTML, JavaScript...
    programmieren. Ich habe schon gehört jQuery (Nachteil: für ein Grid braucht
    man weitere AddOns), Dojo..... [...] 
 
 Content analysis details:   (-1.9 points, 5.0 required)
 
  pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -0.0 RCVD_IN_DNSWL_NONE     RBL: Sender listed at http://www.dnswl.org/, low
                             trust
                             [213.165.64.23 listed in list.dnswl.org]
  0.0 FREEMAIL_FROM          Sender email is freemail (jspranz[at]gmx.de)
 -0.0 SPF_PASS               SPF: sender matches SPF record
  0.0 HTML_MESSAGE           BODY: HTML included in message
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
                             [score: 0.0000]
Status:   


--===============5207375469241522320==
Content-Type: multipart/alternative; boundary=90e6ba6e82fedbfbd604b4890450

--end

I can confirm a similar mistake. 
1. I received lots of cron mails and filter the mails into a folder >1500
2. I delete all messages in the folder
3. I receive new mail
4. Two four days old messages reappear in the folder.

Thomas

Same problem in KDE 4.9.3. Imported old kmail data using the various import tools. Now I have duplicate messages that keep getting readded *every* *time* I switch folders, no matter how many times I delete them, and I get tons of “Maildir '' for collection '' is invalid” and “folder does not exist” errors. This is making it unusable for me.

The type of resource, as listed in akonadiconsole, is “maildir”, by the way.

It looks either your akonadi database is corrupted or something really unexpected happens. Can you check what goes on on the disk? Locate your maildir folder on the disk (see the configuration of your maildir account). Check that the folder and the subfolders are writable for you.
Then check when a new mail appears, what goes on. It should first appear in the "new" folder under "inbox". When you read it, it should be moved to the "cur" folder (and an ",S" appended to the filename). Check if:
- the mail is moved
- the flags is correctly appended

Check also what goes on when you delete a mail (it should be deleted from the disk).

Even more, run "kdebugdialog" in a konsole and disable all debug areas then enable the debug areas for akonadiserver and folder the akonadi_maildir_resourceX (X depends on your system, can be 0,1, etc.), then restart akonadi in that konsole ("akonadictl restart") and save the log what happens when you read a mail and when you delete one.

I don't use maildir. I use postfix with mbox.

After receiving the mails they are definitly gone. It's an akonadi problem.

Thomas: yours is a different issue then. mbox is completely different code from maildir, please report for Akonadi product, MBox resource.

My local folders location shows up as ~/.local/share/local-mail. But there's nothing there except three empty folders, cur, new, and tmp. I do have ability to write to those locations. I do see some sort of directory structure in ~/.local/.local-mail.directory that mirrors my mail folders, though.

All the folders were created using the File > Import messages tool to import from KDE 3's kmail maildirs.

Travis: for you a solution would be to get rid of the current maildir resource and add back again. Just delete the local account from KMail account settings and add back again pointing to the ~/.local/.local-mail.directory. Do it ONLY AFTER you are sure all your mails are indeed in THAT directory.
 That would not tell us what the bug was (for that you'd need to do what I wrote and try to find out where your mails are and what happens with them).
The information of using File->IMport messages is important, thanks for that, we should check how that works (I never used it myself).

I seem to have made a mistake—the alternative folder I mentioned was apparently actually a leftover from back when I installed my current distro and was trying unsuccessfully to get kmail2 working (last modified dates are a long time ago). But I think my real emails are in ~/.local/share/.local-mail.directory (last modified dates a few hours ago). But many directories appear to be missing, particularly the ones I seem to be having trouble with. I tried creating a couple of new directories to see what happens, and those directories did show up in this location. I'll have to do some investigation to see where the messages actually are and what the akonadi debug logs say.

The missing directories in the filesystem are the ones created by the import tool. The only directories that are there are the ones I manually created in kmail2's main GUI. I cannot figure out where in the filesystem the imported email directories are other than their original location used by the old kmail.

Would it work to try manually copying the original maildirs over to the new location? Or has the format changed from kmail to kmail2?

If the old KMail used the standard maildir and not the mix of maildir + mbox folders, copying over should work. 
Still, I'd need some clarifications before you do:  how did you imported your old mails? Folder by folder or pointing to the toplevel mail folder?
You mean that all folders (and mails in them) imported are visible in KMail, but are not in the ~/.local/share/.local-mail.directory ? That sounds like the import failed and the mails are in the akonadi cache, but not on the filesystem (and would explain all kind of weird behavior).

I pointed it to the top level folder and had it import all in one go. The only error I got was regarding one of the tmp directories, but I don't remember which one or what the error was exactly. I got no errors for everything else. Yes, the imported emails are all visible in Kmail2 but apparently not in the directory in the filesystem.

Please, make a control with "akonadi console"
Choose "DB Browser", "collectiontable", "Refresh"
Get a look to the column "remoteId".
Are there any cell(s) with nothing? If yes, let me/us know by copying the raw(s).

Seems I overlooked needing to followup on this somehow—sorry.

In the akonadi console, the only id listed without a remoteid is the one for the “Search” folder. However, despite the fact that I thought I had resolved this before, it appears that *no* new emails have ever been saved to the maildir file system since I performed the import. Which is extremely frustrating because I'm having endless problems with kmail2 to the point of it being unusable, and I'd like some way to import all my mail into another email program.

This bug has only been reported for versions before 4.14, which have been unsupported for at least two years now. Can anyone tell if this bug still present?

If noone confirms this bug for a Framework-based version of kmail2 (version 5.0 or later, as part of KDE Applications 15.12 or later), it gets closed in about three months.

Sorry, but I can't provide info on this one as I stopped using kmail some years ago because of this bug.

Just as announced in my last comment, I close this bug. If you encounter it again in a recent version (at least 5.0 aka 15.08), please open a new one unless it already exists. Thank you for all your input.