Subject: removing dupes from mbox file

From: "Harondel J. Sibble" <[email-masked]>
Date: Sun, 13 Apr 2003 22:10:06 -0700

Anyone know a good tool for removing duplicate messages from an mbox file?
-- 
Harondel J. Sibble 
Sibble Computer Consulting
Creating solutions for the small business and home computer user.
[email-masked] (use pgp keyid 0x3AD5C11D) http://www.pdscc.com
(604) 739-3709 (voice/fax)      (604) 686-2253 (pager)

-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Ben Fowler <[email-masked]>
Date: Mon, 14 Apr 2003 07:55:41 +0100

>Anyone know a good tool for removing duplicate messages from an mbox file?

I am not sure that this is on charter, but asumming that you mean the format
in which mails are concatenated, separated by lines of the  form

>From ???@??? Fri, 4 Oct 2002 1:05:28 -0400

(no :) <URL: http://www.qmail.org/man/man5/mbox.html >
It is quite easy to write a perl script to do this. Here is mine:

#!/usr/bin/perl

use Mac::MoreFiles;
use Digest::MD5 qw(md5_hex);

foreach( @ARGV ) {
	print "Working with $_\n";
	my $fnIN = $_;
	my $fnOUT = "$_.out";
	if( open( IN, "<$fnIN\0" ) ) {
		if( open( OUT, ">$fnOUT" ) ) {
			my @lines;
			my %digests;
			my $md5 = Digest::MD5->new;
			while( <IN> ) {
				if(
				# From ???@??? Fri, 4 Oct 2002 1:05:28 -0400
					/^From \?\?\?\@\?\?\? [A-Za-z]+ [A-Za-z]+ [0-9]+ [0-9][0-9]:[0-9][0-9]:[0-9][0-9] 200[0-9]/  ||
					
				# From ???@??? Wed Oct 16, 2002  10:35:43  AM US/Pacific
					/^From \?\?\?\@\?\?\? [A-Za-z]+ [A-Za-z]+ [0-9]+, 200[0-9] +[0-9]+:[0-9][0-9]:[0-9][0-9]/  ||
					
				# From ???@??? Fri Jan 01 00:23:35 1904
					/^From \?\?\?\@\?\?\? [A-Za-z]+ [A-Za-z]+ [0-9]+ [0-9][0-9]:[0-9][0-9]:[0-9][0-9] 1904/ ||
					
				# From ???@??? Sat, 5 Oct 2002 02:43:18 -0400
					/^From \?\?\?\@\?\?\? +[A-Za-z]+, +[0-9]+ [A-Za-z]+ [0-9][0-9][0-9][0-9] [0-9]+:[0-9][0-9]:[0-9][0-9]/ 
																													) {
					
					if( $#lines > 0 ) {
#						print "Output $#lines lines\n";
						$digest = $md5->hexdigest;
						$md5->reset;
						my $dup = '';
						if( $digests{ $digest } ) {
							$dup = 'dup';
							print "\t$digest $dup\n";
							$#lines = -1;
						} else {
							$digests{ $digest } = 1;
							while( @lines ) {
								my $line = shift @lines;
								print OUT $line;
							}
						}
					};

				} elsif ( /^From \?\?\?\@\?\?\? / ) {
					print "Error: $_";

				};
				
				$md5->add( $_ );
				push( @lines, $_ );
			};
			
			# print last message	
			if( $#lines > 0 ) {
				print "Output $#lines lines\n";
				while( @lines ) {
					print OUT shift @lines;
				}
			};

			print "About to close output file $fnOUT\n";
			close( OUT )
				or print "Error closing file $fnOUT : $!\n";
			MacPerl::SetFileInfo( 'CSOm', 'TEXT', $fnOUT );
			undef %digests;
		} else {
			print "Error opening file $fnOUT for writing: $!\n";
		}
		close( IN )
			or print "Error closing file $fnIN : $!\n";
	} else {
		print "Error opening file '$fnIN' for reading: $!\n";
		next;
	}		
}
-- 
Privileged/Confidential Information may be contained in this message.
If you are not the addressee indicated in this message (or responsible 
for delivery of the message to such person), you may not copy or deliver 
this message to anyone. In such case, you should destroy this message 
and kindly notify the sender by reply email.  Please advise immediately 
if you or your employer do not consent to Internet email messages of this
kind.  Opinions, conclusions and other information in this message that 
do not relate to the official business of ewitness Limited shall be 
understood as neither given nor endorsed by it.

eWitness Limited, Alpha House, Redvers Close, Lawnswood Business Park, Leeds, LS16 6QY
Telephone:  0113 2951112  Fax: 0113 2951114