From 9a69a9aaad35184ed236f415208041bf19e747e3 Mon Sep 17 00:00:00 2001 From: Gerfried Fuchs Date: Wed, 17 Feb 2010 20:23:09 +0100 Subject: [PATCH] Imported Upstream version 0.54.1 --- BUGS | 25 ++ ChangeLog | 346 +++++++++++++++++++++++ README | 54 ++++ TODO | 26 ++ contrib/README.patches | 29 ++ contrib/t-prot-r1.51-debug.diff | 198 ++++++++++++++ contrib/t-prot-r1.51-ftr.diff | 78 ++++++ muttrc.t-prot | 14 + t-prot | 470 ++++++++++++++++++++++++++++++++ t-prot.1 | 307 +++++++++++++++++++++ 10 files changed, 1547 insertions(+) create mode 100644 BUGS create mode 100644 ChangeLog create mode 100644 README create mode 100644 TODO create mode 100644 contrib/README.patches create mode 100644 contrib/t-prot-r1.51-debug.diff create mode 100644 contrib/t-prot-r1.51-ftr.diff create mode 100644 muttrc.t-prot create mode 100755 t-prot create mode 100644 t-prot.1 diff --git a/BUGS b/BUGS new file mode 100644 index 0000000..e97e0b2 --- /dev/null +++ b/BUGS @@ -0,0 +1,25 @@ + KNOWN BUGS + ========== + +For now, please see the TODO file and the BUGS section of the manpage +for known bugs. + + + REPORTING BUGS + ============== + +To report bugs, please drop an email to . + +Please do not report a bug if + * you found it in the TODO file or the BUGS section of the man page. +We do know those and try to fix them as soon as possible. + * you have an old t-prot version. If you encounter a problem, first +see if there is a new t-prot version which fixes the issue. If you upgraded +to the latest version and it *still* occurs, a bug report is just great. + + +If you noticed a bug when processing a message and want to provide the +t-prot team with some useful info, please apply the latest debugging +patch from the contrib/ directory. Please read contrib/README.patches +carefully before applying the patch. + diff --git a/ChangeLog b/ChangeLog new file mode 100644 index 0000000..10d46f7 --- /dev/null +++ b/ChangeLog @@ -0,0 +1,346 @@ +2002-03-23 11:47 jochen + + * t-prot: Fix typo. + +2002-03-22 12:38 jochen + + * t-prot.1: Add ghw to the IDEAS AND INSPIRATION section. + +2002-03-21 23:26 jochen + + * t-prot: Bump to 0.54. + +2002-03-21 14:16 jochen + + * t-prot: More cleanups in the multipart code. + +2002-03-21 13:53 jochen + + * t-prot: Fix comment in the mutt specific multipart code. + +2002-03-21 13:27 jochen + + * t-prot: Fixed assumption in the mutt-specific code that the first + attachment always would be text/plain containing the actual message + text. + +2002-03-20 23:58 jochen + + * t-prot: Fix some false positives in footers and multipart code. + +2002-03-20 18:02 jochen + + * t-prot: More cleanups and performance improvements. Fixed tofu + deletion when diff appended to signature. Fixed broken detection of + mutt(1)'s mime/pgp output. Bumped to 0.53. + +2002-03-20 16:12 jochen + + * t-prot: Make mime/multipart parts with type=message/rfc822 being + processed if the message has been processed by mutt(1) before. + +2002-03-20 14:16 jochen + + * t-prot: Another HUGE cleanup. + +2002-03-20 02:56 jochen + + * t-prot: Major cleanup. Bump to 0.52. + +2002-03-19 15:16 jochen + + * t-prot: Fixed some bugs in the multipart code. Bumped version to + 0.51. + +2002-03-19 13:42 jochen + + * t-prot.1: Add note describing the complicated and confusing line + count thingie when deleting an overlong signature. + +2002-03-19 13:38 jochen + + * t-prot: Correct line count in snipped-overlong-sig message. + +2002-03-18 20:19 jochen + + * muttrc.t-prot, t-prot, t-prot.1: Add new options (-a and -A=dir) + to get rid of those annoying hotmail advertisements appended to + messages from people using gratis email providers. Update manpage + and muttrc sample. + + Add non-usascii-problem to TROUBLESHOOTING section of the man page. + + Bump to v0.50. + +2002-03-18 16:49 jochen + + * t-prot: Fix the line count header. Bump to v0.49. + +2002-03-16 01:01 jochen + + * t-prot.1: Minor fix. + +2002-03-16 00:58 jochen + + * t-prot.1: Minor cleanup. + +2002-03-15 02:48 jochen + + * t-prot: Fix variables initialisation. + +2002-03-14 23:21 jochen + + * t-prot: Fix a bug in the multipart code. Bump to v0.48. + +2002-03-13 09:07 jochen + + * t-prot.1: Add some explanation for the verbatim instruction to + the troubleshooting section. + +2002-03-13 08:52 jochen + + * t-prot: Make verbatim instructions work for everything except + mailing list footers. Bump to v0.47. + +2002-03-13 02:19 jochen + + * t-prot.1: Fixed typo. + +2002-03-12 18:16 jochen + + * t-prot.1: Add TROUBLESHOOTING section. + +2002-03-12 17:44 jochen + + * t-prot.1: Changed one makro which was overseen last commit. + +2002-03-12 17:24 jochen + + * t-prot.1: Big man page cleanup. + +2002-03-12 16:41 jochen + + * t-prot.1: Further improvement of the -L man page text. + +2002-03-12 16:39 jochen + + * t-prot.1: Improved man page text for -L. + +2002-03-12 16:37 jochen + + * t-prot.1: Clarify the man page text for -l and -L. + +2002-03-12 03:47 jochen + + * t-prot, t-prot.1: Add the "tolerate diff" option. Bump to v0.46. + +2002-03-09 18:18 jochen + + * muttrc.t-prot, t-prot, t-prot.1: Add option -M/--mua and try to + detect attachments only when necessary. Updated man page and + muttrc sample. Version bump to 0.45. + +2002-03-08 21:29 jochen + + * t-prot: Fix an off-by-one error when removing mutt(1)'s + attachments. Credits to Gerfried Fuchs for reporting and locating + the bug. + +2002-03-02 03:26 jochen + + * t-prot: Some minor cleanups. + +2002-03-02 03:18 jochen + + * t-prot: Some minor cleanups. + +2002-03-01 11:10 jochen + + * t-prot: Fixed a missing range check reported by Gerfried Fuchs + with messages only containing the mailing list footer. + +2002-02-26 13:20 jochen + + * t-prot: Add infrastructure for better debugging. + +2002-02-26 12:11 jochen + + * muttrc.t-prot: Make the muttrc example a little shorter, and add + colors. + +2002-02-26 04:05 jochen + + * t-prot: Improve empty line handling between body and signature. + +2002-02-26 03:50 jochen + + * t-prot: Fixed thinko in signature handling. Be more tolerant with + empty lines around signatures and mailing list footers. + + Version bump to 0.42. + +2002-02-25 20:20 jochen + + * t-prot: Widened the rexexp for mutt(1)'s attachments a little - + some cases were not caught before. + +2002-02-25 19:48 jochen + + * t-prot: Gerfried Fuchs found a bug in the signature code (many + thanks!). It was ugly anyway, so that part got a clean rewrite. + + Bumped version to 0.41. + +2002-02-22 16:55 jochen + + * t-prot: Another slight performance hit with multipart messages. + +2002-02-22 16:36 jochen + + * t-prot: Leave only the body of the first multipart message part + for processing, not the headers. + +2002-02-22 00:27 jochen + + * t-prot.1: Updated the manpage to reflect the fix of the multipart + bug. + +2002-02-22 00:24 jochen + + * t-prot: Fix no TOFU is detected in multipart messages without + signature and without previous preparation by mutt(1). Now + multipart messages as described in RFC 1341 are detected properly. + Only the first part is processed, and only if it has the content + type "text/plain" or none specified as described in RFC 1341. + +2002-02-20 17:51 jochen + + * t-prot: Bump version to 0.39. + +2002-02-20 17:50 jochen + + * t-prot.1: Updated manpage for long options. + +2002-02-18 20:44 jochen + + * t-prot.1: Updated credits in manpage. + +2002-02-16 01:06 jochen + + * muttrc.t-prot, t-prot, t-prot.1: Switched command line system to + Getopt::Mixed as recommended by Gerfried Fuchs. One of the reasons + is that now the parameter of the option '-S' is optionally, what + was not cleanly realizeable before. The opportunity has been used + to do a cleanup on some things, too ... + + Please note that Getopt::Mixed now is mandatory to run t-prot - you + can get it at CPAN. The trade-off seems worthy IMHO - please drop + an email to if you do not think so, so the + matter can be diskussed. :) + + Updated manpage. + +2002-02-15 04:49 jochen + + * t-prot.1: Beautify the manpage, and add some more info about bugs + and todos. + +2002-02-15 04:30 jochen + + * t-prot: Beautify help text. + +2002-02-14 17:32 jochen + + * t-prot, t-prot.1: Bjoern Buerger requested an option to make + t-prot write a syslog entry for each message bounced when invoked + with -p. So -d is new. Will only write syslog entries if + Sys::Syslog is available but will not crash if not. + +2002-02-14 14:26 jochen + + * t-prot: Minor cleanup. + +2002-02-14 12:04 jochen + + * t-prot: Fix unbelieveable stupid bug in attachment fix. Bump to + v0.36. + +2002-02-13 02:08 jochen + + * t-prot: Version bump to 0.35. + +2002-02-12 14:57 jochen + + * t-prot, t-prot.1: Added Option -P=MESSAGE. Updated manpage. + +2002-02-12 13:47 jochen + + * t-prot.1: Update manpage for -S=n. + +2002-02-12 13:45 jochen + + * muttrc.t-prot, t-prot: Improve bounce message. Change option -S + to accept an integer denoting the maximum allowed lines for a valid + signature. + +2002-02-12 05:44 jochen + + * t-prot, t-prot.1: New option -p=ADDRESS added: Finally you may + use t-prot in your MTA's aliases file to bounce TOFUed emails. + Manpage updated. + +2002-02-12 02:03 jochen + + * t-prot: Corrected help when called with -h. Small indent cleanup. + +2002-02-11 21:40 jochen + + * t-prot.1: Updated manpage date. + +2002-02-11 21:33 jochen + + * t-prot, t-prot.1: Gerfried Fuchs reported that t-prot eats up + attachments when hiding lines, that flaw should be fixed right now. + Manpage has been updated to reflect upon that fact. + +2002-02-11 20:13 jochen + + * t-prot: Corrected line count for overlong signatures. Problem + reported by Gerfried Fuchs (thanks, Alfie!). + +2002-01-29 15:57 jochen + + * t-prot: Remove goto. Patch by Gerfried Fuchs . + +2002-01-28 21:34 jochen + + * t-prot: Bump version to 0.32. + +2002-01-28 17:53 jochen + + * t-prot: Just again a version fix. Should be done now... + +2002-01-28 17:52 jochen + + * t-prot: Sigh. Another version fix. + +2002-01-28 17:50 jochen + + * t-prot.1: Fix manpage version. + +2002-01-28 17:48 jochen + + * t-prot, t-prot.1: Fix version strings. + +2002-01-28 17:46 jochen + + * muttrc.t-prot, t-prot, t-prot.1: Version cleanup. + +2002-01-28 17:39 jochen + + * muttrc.t-prot, t-prot, t-prot.1: Initial revision + +2002-01-28 17:39 jochen + + * muttrc.t-prot, t-prot, t-prot.1: Initial version. + diff --git a/README b/README new file mode 100644 index 0000000..3636629 --- /dev/null +++ b/README @@ -0,0 +1,54 @@ + t-prot -- TOFU protection + ========================= + +1. What the hell is TOFU? + As the man page says: + TOFU is an abbreviation which mixes German and English words; it expands + to "text oben, full-quote unten" which means "text above - full quote + below" and describes the style of so many users who let their mailer or + newsreader quote everything of the previous message and just add some + text at the top; obviously they think that quoted text must not be + changed at all. This is quite annoying as it needlessly sends a lot of + data even when it is not required. Some editing of messages is desired. + Please point these people to the page http://learn.to/edit_messages - + thank you! + +2. What does the script do? + It detects, and when demanded hides annoying parts in rfc822 messages: + TOFU, signatures (especially when they are too long), excessive + punktuation, blocks of empty lines, trailing spaces and tabs. + For use inside of MTAs or MDAs it may exit with appropriate libc exit + codes, so annoying messages may be bounced easily. + +3. For what can I use it? + There are several possibilities. One is to filter your email or news + messages when displaying them in your MUA. Another is blocking annoying + messages entirely from your system - using the script in some sendmail + or procmail rule, or perhaps even inside innd. + +4. Give me some example! What about an example configuration for the MUA + mutt(1)? + An example is included in the distribution. Please see the man page for + further details on the activated features. + +5. And what about other MUAs? + I just use mutt, so I do not know how to filter messages in other MUAs. + If you know how to incorporate t-prot e.g. in Gnus, please just drop me + a note. ;) + +6. And what about an example for bouncing emails? + Put a line like the following in your /etc/mail/aliases: + ------SNIPP------ + notofu: |"/usr/local/bin/t-prot -cemtS -p=user@mydomain" + ------SNIPP------ + (Do not forget to call `newaliases`.) Messages for notofu@mydomain will + be scanned for TOFU. If t-prot found TOFU in a message, it will bounce, + otherwise it will be forwarded to user@mydomain. This works great with + sendmail, if you use another MTA you probably have to adapt it. + +7. Where did the idea come from? + Many thanks to Gerhard H. Wrodnigg who uses a TOFU protection script + in order to keep the responses to his cancel bot reasonably short. The + entire inspiration for this hack came from the "TOFU protection" line of + his script on many usenet postings. + diff --git a/TODO b/TODO new file mode 100644 index 0000000..16f6fea --- /dev/null +++ b/TODO @@ -0,0 +1,26 @@ + TODO LIST FOR T-PROT + ==================== + +Ordered by priority (highest comes first): + + * Fix internationalization bug with mutt(1) reported by Christian +Borss: If you use mutt with locales for languages other than English, +most probably detection of mutt's multipart lines (attachments, pgp +validation, ...) will fail. A workaround is to call mutt with LC_ALL=C +for now (OK, you won't have localization then, but at least t-prot works +fine, and it will be fixed ASAP anyway). + + * Add an option to repair "Kammquotings". + + * Better heuristics for M$ style TOFU. The actual algorithm is very +greedy (actually, it simply snipps everything down to the signature). +Perhaps add an option to toggle between both behaviours. Comments welcome. + + +For more issues and/or some details, please see the BUGS section of the +manpage. + +To report bugs, please first see the BUGS file. +To make suggestions or tell about features you miss, please drop an email +to . + diff --git a/contrib/README.patches b/contrib/README.patches new file mode 100644 index 0000000..492c756 --- /dev/null +++ b/contrib/README.patches @@ -0,0 +1,29 @@ +t-prot-*-debug.diff +=================== +With this patch applied t-prot dumps a lot of debugging output to +"/tmp/t-prot-d_$PID_*". +You might want to use the command line parameter "--debugpath" to change +the storage location for this data, so the data (which contains the +entire message you process) will not be stored where other users have +read/write permissions (which would otherwise compromise confidential +messages you might process). +Please be careful not to apply this patch for daily use. It is solely +intended for easier providing debug data when reporting bugs to the +t-prot team (if you encounter any bugs, please drop a message to + -- thanks!). + + +t-prot-*-ftr.diff +================= +With this patch applied t-prot makes footer detection really greedy: We +assume that commercial email providers aren't even frightened to append +changing texts *under* their ads which are appended to the message body. +Because these texts even have changing *lengths* we simply detect the +lines of the footer *anywhere* in the body of the message and assume that +everything below belongs to the footer. (Man, if life where always that +easy! ;) +With this patch even GMX ads should be easy to hide -- you buy this with +a slight performance hit (which is the reason the patch hasn't made it +into the vanilla tree yet), and with the possibility that sometimes the +algorithm is just a little *too* greedy. + diff --git a/contrib/t-prot-r1.51-debug.diff b/contrib/t-prot-r1.51-debug.diff new file mode 100644 index 0000000..b18c8ee --- /dev/null +++ b/contrib/t-prot-r1.51-debug.diff @@ -0,0 +1,198 @@ +--- t-prot 2002-03-25 02:15:50.000000000 +0100 ++++ t-prot-d 2002-03-28 16:27:02.000000000 +0100 +@@ -9,12 +9,13 @@ + $EX_OK $EX_USAGE $EX_DATAERR $EX_UNAVAILABLE $EX_BOUNCE + $ad $ads $boun $cr $diff $elli $footers $hdrs $indent $lsig $maxsig + $mda $ml $ms $mua $ofile $sendmail $sig $sysl $trad $trsp ++ $dc $debugpath + ); + + + # Version info + $VER = '0.54'; +-$REV = ''; ++$REV = '-debug'; + $REL = q$Revision: 1.51 $; chop($REL); + # From + # (you might have to adjust those if not using GNU libc) +@@ -29,6 +30,10 @@ + $sendmail = '/usr/sbin/sendmail -oi'; # MTA expecting mail on STDIN + $boun = "Blocked by $0: This user does not accept TOFUed email. Please see and for more info. Have a nice day!\n"; + $ofile = '-'; # use STDOUT if nothing is specified ++$dc = 0; # debug dump counter ++$debugpath = '/tmp'; ++ # this should better be changed by command line for ++ # confidential messages + # end of user adjusted vals + + +@@ -39,6 +44,7 @@ + -A=DIRECTORY ad footer directory, treat ad footers as signature + -c merge multiple blank lines + -d, --debug print notice to syslog when bouncing; requires -p ++ --debugpath=DIR where to dump debug info + --diff tolerate diffs appended *after* the signature + -e force ellipsis for excessive punctuation + -h, --help show this short help and exit +@@ -64,9 +70,30 @@ + # version(): print version info and exit with appropriate exit code + sub version { + print "$0 v$VER$REV ($REL), Jochen Striepe ++This is a debugging version. DO NOT INSTALL THIS VERSION FOR DAILY USE! + Get the latest version at \n"; + exit($EX_OK); + } ++ ++# write useful debugging output ++# usage: debug_dump('/tmp/foo', $foo, $bar, @klotz); ++sub debug_dump { ++ if (scalar(@_)<2) { die "bad parameters"; } ++ my $f = shift; ++ if (-l $f) { die "We won't write to $f for security reasons"; } ++ ++ open(DBG, ">$f") || die "Could not open $f: $!"; ++ my $foo; ++ while (@_) { ++ $foo = shift; ++ chomp($foo); ++ $^W = 0; ++ print DBG "$foo\n"; ++ $^W = 1; ++ } ++ close DBG; ++} ++ + + # remove_footers(): remove any trailing appearance of footers contained + # in the given directory. +@@ -139,6 +166,7 @@ + for ($x=0; $x<$#$lines; $x++) { if (@$lines[$x] =~ /^$/) { last; }; } + @hdr = @$lines[0..$x]; + splice(@$lines, 0, $x+1); ++ debug_dump($debugpath.'_'.$$.'_'.$dc++, @$lines); + + # See if we have a multipart content type. If yes, see if it is already + # ripped (e.g. by mutt(1)), otherwise only leave the first part if it +@@ -240,6 +268,7 @@ + last; + } + } ++ debug_dump($debugpath.'_'.$$.'_'.$dc++, @$lines); + + # Pipe message/rfc822 parts to another instance of process_msg() + # for further processing. +@@ -267,12 +296,15 @@ + } + } + } ++ debug_dump($debugpath.'_'.$$.'_'.$dc++, @$lines); + + # Remove ML footers: + remove_footers($lines, \@ftr, $footers, undef); ++ debug_dump($debugpath.'_'.$$.'_'.$dc++, @$lines); + + # Remove ad footers: + remove_footers($lines, \@ads, $ads, undef); ++ debug_dump($debugpath.'_'.$$.'_'.$dc++, @$lines); + + # Remove signature: + if (scalar(@$lines)) { +@@ -307,6 +339,7 @@ + } + } + } ++ debug_dump($debugpath.'_'.$$.'_'.$dc++, @$lines); + + # Now care about TOFU. + # One common mispractice is M$ style TOFU: +@@ -331,6 +364,7 @@ + $j = scalar(@$lines)-$x; + splice(@$lines, $x); + } ++ debug_dump($debugpath.'_'.$$.'_'.$dc++, @$lines); + + # Nothing? Then try traditional TOFU: + if ($trad && (!$j) && !$vrb[$#$lines]) { +@@ -345,6 +379,7 @@ + pop(@$lines); + } + } ++ debug_dump($debugpath.'_'.$$.'_'.$dc++, @$lines); + + # OK, if we found TOFU, we will leave a message that we were here... + if ($j) { +@@ -368,6 +403,7 @@ + + push(@$lines, "[---=| TOFU protection by $0: $j lines snipped |=---]\n"); + } ++ debug_dump($debugpath.'_'.$$.'_'.$dc++, @$lines); + + + # Care for trailing whitespaces: +@@ -376,6 +412,7 @@ + if (!$vrb[$x]) { $$lines[$x] =~ s/[\ \t]+$//; } + } + } ++ debug_dump($debugpath.'_'.$$.'_'.$dc++, @$lines); + + # Care for punctuation abuse: + if ($elli) { +@@ -383,6 +420,7 @@ + if (!$vrb[$x]) { $$lines[$x] =~ s/([.?!])(\1{2})\1+/$1 . $2/eg; } + } + } ++ debug_dump($debugpath.'_'.$$.'_'.$dc++, @$lines); + + # (Nearly) at last care for multiple blank lines. (Do not do this + # earlier -- the way it is done right now would screw up the verbatim +@@ -396,6 +434,7 @@ + else { $t = 0; } + } + } ++ debug_dump($debugpath.'_'.$$.'_'.$dc++, @$lines); + + # Everything changing the body is done now. Time to fix the line count + # header so naive clients do not get confused. Just to be sure, append +@@ -428,12 +467,13 @@ + # get command line params: + $0 =~ s!^.*/!!; + Getopt::Mixed::init('a A=s c d e h i=s L=s l m M=s o=s P=s p=s r S:i'. +- ' s t v w debug>d diff help>h mua>M version>v'); ++ ' s t v w debug>d debugpath diff help>h mua>M version>v'); + while (my ($opt, $val, $pretty) = nextOption()) { + if ($opt eq 'a') { $ad = 1; } + elsif ($opt eq 'A') { $ads = $val; } + elsif ($opt eq 'c') { $cr = 1; } + elsif ($opt eq 'd') { $sysl = 1; } ++ elsif ($opt eq 'debugpath') { $debugpath = $val; } + elsif ($opt eq 'diff') { $diff = 1; } + elsif ($opt eq 'e') { $elli = 1; } + elsif ($opt eq 'i') { $ifile = $val; } +@@ -454,17 +494,26 @@ + } + Getopt::Mixed::cleanup(); + if (($ml && $footers eq '')||($ad && $ads eq '')) { help(); } +- ++$debugpath .= '/t-prot-d'; + + # Read message: + open(IN, $ifile) || die "Could not open $ifile: $!"; + my @message = ; + close IN; ++debug_dump($debugpath.'_'.$$.'_'.$dc++, ++ $VER.$REV, $REL, ++ "ad=$ad", "ads=$ads", "cr=$cr", "sysl=$sysl", "elli=$elli", "ml=$ml", ++ "ms=$ms", "mda=$mda", "mua=$mua", "hdrs=$hdrs", "lsig=$lsig", "sig=$sig", ++ "trad=$trad", "trsp=$trsp", "footers=$footers", "ifile=$ifile", ++ "ofile=$ofile", "maxsig=$maxsig", "indent=$indent", "sendmail=$sendmail", ++ "boun=$boun"); ++debug_dump($debugpath.'_'.$$.'_'.$dc++, @message); + + # this should be self-explanatory: + process_msg(\@message); + + # Finally, print clean lines: + write_msg(($mda?"|$sendmail $mda":">$ofile"), \@message); ++debug_dump($debugpath.'_'.$$.'_'.$dc++, @message); + + # eof diff --git a/contrib/t-prot-r1.51-ftr.diff b/contrib/t-prot-r1.51-ftr.diff new file mode 100644 index 0000000..a11af29 --- /dev/null +++ b/contrib/t-prot-r1.51-ftr.diff @@ -0,0 +1,78 @@ +--- t-prot-x 2002-03-25 02:15:50.000000000 +0100 ++++ t-prot 2002-03-25 03:24:03.000000000 +0100 +@@ -14,7 +14,7 @@ + + # Version info + $VER = '0.54'; +-$REV = ''; ++$REV = '-ftr'; + $REL = q$Revision: 1.51 $; chop($REL); + # From + # (you might have to adjust those if not using GNU libc) +@@ -75,6 +75,7 @@ + my $S = shift; # array to store removed lines in + my $F = shift; # footers dir name + my $O = shift; # remove only one footer? ++ my $V = shift; # allow footers match before end of message + + if ($F && scalar(@$L)) { + opendir(DIR, $F) || die "Could not open $F: $!"; +@@ -85,25 +86,27 @@ + open(IN, "$F/$f") || die "Could not open $F/$f: $!"; + my @l = ; + close IN; ++ for (my $z=0; $z<=$#l; $z++) { chomp($l[$z]); } + +- while (scalar(@l)<=scalar(@$L)) { +- my $y = 0; +- for(my $x=1; $x<=scalar(@l); $x++) { +- chomp($l[scalar(@l)-$x]); +- if (index($$L[scalar(@$L)-$x], $l[scalar(@l)-$x])!=0) { +- $y = 1; +- } +- } +- if (!$y) { +- unshift(@$S, @$L[$#$L-$#l..$#$L]); +- splice(@$L, $#$L-$#l); +- while (scalar(@$L) && $$L[$#$L] =~ /^\s*$/) { +- unshift(@$S, pop(@$L)); +- } +- if ($O) { last; } +- } +- else { last; } +- } ++ WIPE: for (my $z=$V?(scalar(@$L)-scalar(@l)):0; $z>=0; $z--) ++ { ++ if (scalar(@l)+$z<=scalar(@$L)) { ++ my $y = 0; ++ for(my $x=1; $x<=scalar(@l); $x++) { ++ if (index($$L[scalar(@$L)-$x-$z], $l[scalar(@l)-$x])!=0) { ++ $y = 1; ++ } ++ } ++ if (!$y) { ++ unshift(@$S, @$L[$#$L-$#l-$z..$#$L]); ++ splice(@$L, $#$L-$#l-$z); ++ while (scalar(@$L) && $$L[$#$L] =~ /^\s*$/) { ++ unshift(@$S, pop(@$L)); ++ } ++ if ($O) { last; } else { goto WIPE; } ++ } ++ } ++ } + } + } + } +@@ -269,10 +272,10 @@ + } + + # Remove ML footers: +- remove_footers($lines, \@ftr, $footers, undef); ++ remove_footers($lines, \@ftr, $footers, undef, undef); + + # Remove ad footers: +- remove_footers($lines, \@ads, $ads, undef); ++ remove_footers($lines, \@ads, $ads, undef, '1'); + + # Remove signature: + if (scalar(@$lines)) { diff --git a/muttrc.t-prot b/muttrc.t-prot new file mode 100644 index 0000000..dacddf8 --- /dev/null +++ b/muttrc.t-prot @@ -0,0 +1,14 @@ +# $Id: muttrc.t-prot,v 1.7 2002/03/18 19:19:23 jochen Exp $ +set display_filter='t-prot -acelmtS -Mmutt -L$HOME/.mutt/mlfooters -A$HOME/.mutt/adfooters' + +# toggle TOFU protection with ESC-0 and ESC-1 +macro generic \e0 ":unset display_filter\n" "Turn TOFU protection off" +macro generic \e1 ":set display_filter='t-prot -acelmtS -Mmutt -L$HOME/.mutt/mlfooters -A$HOME/.mutt/adfooters'\n" "Turn TOFU protection on" + +# same in pager mode - ugly but what the hell... +macro pager \e0 ":unset display_filter; exec exit\n:exec display-message\n" "Turn TOFU protection off" +macro pager \e1 ":set display_filter='t-prot -acelmtS -Mmutt -L$HOME/.mutt/mlfooters -A$HOME/.mutt/adfooters'; exec exit\n:exec display-message\n" "Turn TOFU protection on" + +# highlight TOFU protection: +color body brightmagenta black "^\\[---.*" +color body green black "^#v[-+]" diff --git a/t-prot b/t-prot new file mode 100755 index 0000000..7cd2da4 --- /dev/null +++ b/t-prot @@ -0,0 +1,470 @@ +#!/usr/bin/perl -w +# $Id: t-prot,v 1.51 2002/03/23 10:47:32 jochen Exp $ + +require 5.005; +use strict; +use Getopt::Mixed qw(nextOption); +use vars qw( + $VER $REV $REL + $EX_OK $EX_USAGE $EX_DATAERR $EX_UNAVAILABLE $EX_BOUNCE + $ad $ads $boun $cr $diff $elli $footers $hdrs $indent $lsig $maxsig + $mda $ml $ms $mua $ofile $sendmail $sig $sysl $trad $trsp +); + + +# Version info +$VER = '0.54'; +$REV = ''; +$REL = q$Revision: 1.51 $; chop($REL); +# From +# (you might have to adjust those if not using GNU libc) +$EX_OK = 0; +$EX_USAGE = 64; +$EX_DATAERR = 65; +$EX_UNAVAILABLE = 69; +$EX_BOUNCE = $EX_UNAVAILABLE; +# Please adjust these vals to your needs: +$maxsig = 4; # max. valid signature length +$indent = '>'; # Indent string, regexp to identify a quoted line +$sendmail = '/usr/sbin/sendmail -oi'; # MTA expecting mail on STDIN +$boun = "Blocked by $0: This user does not accept TOFUed email. Please see and for more info. Have a nice day!\n"; +$ofile = '-'; # use STDOUT if nothing is specified +# end of user adjusted vals + + +# help(): print help text and exit with appropriate exit code +sub help { + print "Usage: $0 [options] + -a remove ad footers; requires -A + -A=DIRECTORY ad footer directory, treat ad footers as signature + -c merge multiple blank lines + -d, --debug print notice to syslog when bouncing; requires -p + --diff tolerate diffs appended *after* the signature + -e force ellipsis for excessive punctuation + -h, --help show this short help and exit + -i=INFILE file to be read; '-' for STDIN (default) + -L=DIRECTORY mailling list footer directory, treat mailing list + footers as signature + -l delete mailing list footer; requires -L + -M, --mua=MUA turn on special treatment for some mail user agents + -m delete MS style TOFU; careful: might be too agressive + -o=OUTFILE file to be written to; '-' for STDOUT (default) + -P=MESSAGE user defined bounce message; requires -p + -p=ADDRESS redirect to ADDRESS if no TOFU was found + -r delete mail header lines + -S[=n] supress signatures with more than n lines; + default is $maxsig if n not specified + -s delete signature + -t delete traditional style TOFU + -v, --version show version string and exit + -w delete trailing whitespaces\n"; + exit($EX_USAGE); +} + +# version(): print version info and exit with appropriate exit code +sub version { + print "$0 v$VER$REV ($REL), Jochen Striepe +Get the latest version at \n"; + exit($EX_OK); +} + +# remove_footers(): remove any trailing appearance of footers contained +# in the given directory. +sub remove_footers { + my $L = shift; # array of message lines + my $S = shift; # array to store removed lines in + my $F = shift; # footers dir name + my $O = shift; # remove only one footer? + + if ($F && scalar(@$L)) { + opendir(DIR, $F) || die "Could not open $F: $!"; + my @feet = grep { /^[^.]/ && -f "$F/$_" } readdir DIR; + closedir DIR; + + foreach my $f (@feet) { + open(IN, "$F/$f") || die "Could not open $F/$f: $!"; + my @l = ; + close IN; + + while (scalar(@l)<=scalar(@$L)) { + my $y = 0; + for(my $x=1; $x<=scalar(@l); $x++) { + chomp($l[scalar(@l)-$x]); + if (index($$L[scalar(@$L)-$x], $l[scalar(@l)-$x])!=0) { + $y = 1; + } + } + if (!$y) { + unshift(@$S, @$L[$#$L-$#l..$#$L]); + splice(@$L, $#$L-$#l); + while (scalar(@$L) && $$L[$#$L] =~ /^\s*$/) { + unshift(@$S, pop(@$L)); + } + if ($O) { last; } + } + else { last; } + } + } + } +} + +# write_msg(): output +sub write_msg { + my $O = shift; + my $l; + + open(OUT, $O) || die "Could not open $O: $!"; + while (scalar(@_)) { + $l = shift; + if (defined $l) { + $^W = 0; + print OUT @$l; + $^W = 1; + } + } + close OUT; +} + +# process_msg(): This one proc does *everything* what has to be done with +# the lines of the message +sub process_msg { + my $lines = shift; + + my ($j, $x, $verb) = (0, 0, 0); + my (@ads, @hdr, @bo1, @bo2, @ftr, @sig, @vrb, @att) = + ((), (), (), (), (), (), (), (), ()); + + # First, remove and store lines we might need later... + # Remove headers: + for ($x=0; $x<$#$lines; $x++) { if (@$lines[$x] =~ /^$/) { last; }; } + @hdr = @$lines[0..$x]; + splice(@$lines, 0, $x+1); + + # See if we have a multipart content type. If yes, see if it is already + # ripped (e.g. by mutt(1)), otherwise only leave the first part if it + # is plain text (if not, we are done - non-text messages are not our + # business). + if (lc($mua) ne 'mutt') { + for ($x=0; $x=scalar(@$lines)) { exit($EX_DATAERR); } + + if ($bar =~ /^text\/plain/i) { + my $z; + for ($z=1; $x+$z<@$lines; $z++) { + if (index($$lines[$x+$z], '--'.$foo)==0) { + last; + } + } + if ($x+$z>=scalar(@$lines)) { exit($EX_DATAERR); } + + @bo2 = @$lines[$x+$z..$#$lines]; + splice(@$lines, $x+$z); + if ($$lines[$#$lines] =~ /^\s*$/) { + unshift(@bo2, pop @$lines); + } + @bo1 = @$lines[0..$x]; + splice(@$lines, 0, $x+1); + last; + } + else { + write_msg(($mda?"|$sendmail $mda":">$ofile"), + ($hdrs?undef:\@hdr), $lines); + exit; + } + } + } + } + last; + } + } + } + + + # Protect verbatims: + $verb = 0; + for ($x=0; $x[]*\[-- Attachment #(\d+)(: .*)? --\]\s*$/ && + (($1 ne '1') || + ($x[]*\[-- Type: text\/plain/))) || + ($$lines[$x] =~ /^[^>[]*\[-- End of .* data --\]\s*$/)) + { + @att = @$lines[$x..$#$lines]; + splice(@$lines, $x); + if (scalar(@$lines) && $$lines[$#$lines] =~ /^\s*$/) { + unshift(@att, pop(@$lines)); + } + last; + } + } + + # Pipe message/rfc822 parts to another instance of process_msg() + # for further processing. + # Please note that we cannot see what a hierarchy the original + # message had -- if there were message/rfc822 parts within other + # message/rfc822 parts constellations can occur which we cannot + # resolve. Therefore we simply do not even try to be smart. This + # should work for most situations: + if (scalar(@att)) { + for ($x=0; $x<$#att; $x++) { + if ($vrb[scalar(@$lines)+$x]) { next; } + # The following regexp is quite ugly because for most + # users the line is coloured using termcap... (bah!) + if ($att[$x]=~/^[^>[]*\[-- Attachment #\d+(: .*)? --\]\s*$/ && + $att[$x+1] =~ /^[^>[]*\[-- Type: message\/rfc822/) + { + $x += 2; + while ($att[$x] !~ /^\s*$/) { $x++; } + $x++; + + my @tmp = @att[$x..$#att]; + process_msg(\@tmp); + splice(@att, $x, scalar(@att)-$x, @tmp); + } + } + } + } + + # Remove ML footers: + remove_footers($lines, \@ftr, $footers, undef); + + # Remove ad footers: + remove_footers($lines, \@ads, $ads, undef); + + # Remove signature: + if (scalar(@$lines)) { + for ($x=0; $x$lsig))) { + if ($lsig && !$sig) { + push(@sig, "[---=| Overlong signature removed by $0: " . + (scalar(@$lines)-$x) . " lines snipped |=---]\n"); + } + splice(@$lines, $x); + } + elsif ($#$lines-$x<=($lsig?$lsig:$maxsig)) { + @sig = @$lines[$x..$#$lines]; + splice(@$lines, $x); + } + last; + } + } + } + + # Now care about TOFU. + # One common mispractice is M$ style TOFU: + if ($ms) { + # bloat this array if you want more internationalization: + my @tofu = ('Original Message', + 'Ursprüngliche Nachricht', + 'Ursprungliche Nachricht', + 'Mensagem original'); + + DONE: for ($x=0; $x=0; $x--) { + if ((!$vrb[$x]) && $$lines[$x] =~ /^\s*$/) { + if ($t<2) { $t++; } else { splice(@$lines, $x, 1); } + } + else { $t = 0; } + } + } + + # Everything changing the body is done now. Time to fix the line count + # header so naive clients do not get confused. Just to be sure, append + # the old line count to X-headers. + my $l = scalar(@bo1) + scalar(@$lines) + scalar(@att) + scalar(@bo2) + + (!$sig?scalar(@sig):0) + (!$ml?scalar(@ftr):0) + + (!$ad?scalar(@ads):0); + for ($x=0; $xd diff help>h mua>M version>v'); +while (my ($opt, $val, $pretty) = nextOption()) { + if ($opt eq 'a') { $ad = 1; } + elsif ($opt eq 'A') { $ads = $val; } + elsif ($opt eq 'c') { $cr = 1; } + elsif ($opt eq 'd') { $sysl = 1; } + elsif ($opt eq 'diff') { $diff = 1; } + elsif ($opt eq 'e') { $elli = 1; } + elsif ($opt eq 'i') { $ifile = $val; } + elsif ($opt eq 'L') { $footers = $val; } + elsif ($opt eq 'l') { $ml = 1; } + elsif ($opt eq 'm') { $ms = 1; } + elsif ($opt eq 'M') { $mua = $val; } + elsif ($opt eq 'o') { $ofile = $val; } + elsif ($opt eq 'P') { $boun = $val; } + elsif ($opt eq 'p') { $mda = $val; } + elsif ($opt eq 'r') { $hdrs = 1; } + elsif ($opt eq 'S') { $lsig = $val ? $val : $maxsig; } + elsif ($opt eq 's') { $sig = 1; } + elsif ($opt eq 't') { $trad = 1; } + elsif ($opt eq 'v') { version(); } + elsif ($opt eq 'w') { $trsp = 1; } + else { help(); } +} +Getopt::Mixed::cleanup(); +if (($ml && $footers eq '')||($ad && $ads eq '')) { help(); } + + +# Read message: +open(IN, $ifile) || die "Could not open $ifile: $!"; +my @message = ; +close IN; + +# this should be self-explanatory: +process_msg(\@message); + +# Finally, print clean lines: +write_msg(($mda?"|$sendmail $mda":">$ofile"), \@message); + +# eof diff --git a/t-prot.1 b/t-prot.1 new file mode 100644 index 0000000..ce462b4 --- /dev/null +++ b/t-prot.1 @@ -0,0 +1,307 @@ +.\" Copyright Jochen Striepe , 2001-2002. +.\" License see below. +.\" +.\" $Id: t-prot.1,v 1.30 2002/03/22 11:38:35 jochen Exp $ +.\" +.TH T-PROT "1" "March 2002" "T-PROT" +.SH NAME +t-prot \- TOFU Protection - Display Filter for RFC822 messages +.SH SYNOPSIS +.BI "t-prot [" OPTIONS "]..." +.SH DESCRIPTION +.PP +This program is a filter which shall improve the readability +for messages (emails and posts) by *hiding* some annoying parts, +e.g. mailing list footers, signatures, and TOFU (see definition below), +as well as squeezing sequences of blank lines or punctuation. +.br +The filter is written in Perl and relies on input to be a single RFC822 +conform message. +.PP +For easy usage, you can download a file to be included in your ~/.muttrc: +.IR http://www.escape.de/users/tolot/mutt/t-prot/muttrc.t-prot . +.SH OPTIONS +.TP +.BR "\-A" =DIRECTORY +"ad footer directory": +Defines the directory which contains the advertisement list footers (one +footer per file) which are to be tested when removing them with option +.BR -a . +.br +This option is also needed if you do not want signature lengths to be +counted wrong or fullquotes get undetected when an ad footer is +appended at the bottom of the message (especially when using +.B "\-S" +or +.BR "\-t" ). +.TP +.B "\-a" +"commercial signature": +Hides "footers" (signatures) from commercial email providers. +.br +.IR NOTE : +This requires a directory with footer files to be given with option +.BR "\-A" =DIRECTORY. +.TP +.B "\-c" +"compress": +Squeezes a sequence of blank lines to just one blank line. +.TP +.B "\-d, \-\-debug" +"debug": +Print envelope info to syslog when bouncing TOFU contaminated email. +Default syslog facility is mail.debug. Requires +.BR \-p . +.TP +.B "\-\-diff" +"tolerate diff": +Tolerate unified diff (see +.BR diff (1) +and +.BR patch (1)) +appended *after* the signature (which usually makes the signature too long +to be detected). Not entirely consequent but sometimes useful, e.g. at the +linux kernel mailing list. +.TP +.B "\-e" +"ellipsis": +Squeezes a sequence of four or more dots, exclamation marks, or question marks +to only three dots or marks, respectively. +.TP +.B "\-h, \-\-help" +"help": +Displays a short help text with a summary on all options, and exits. +.TP +.BR "\-i" =FILE +"input file": +Defines a file for input; the default input is from '-' i.e. STDIN. +.TP +.BR "\-L" =DIRECTORY +"list footer directory": +Defines the directory which contains the mailing list footers (one footer +per file) which are to be tested when removing them with option +.BR -l . +.br +This option is also needed if you do not want signature lengths to be +counted wrong or fullquotes get undetected when a mailing list footer is +appended at the bottom of the message (especially when using +.B "\-S" +or +.BR "\-t" ). +.TP +.B "\-l" +"list signature": +Hides "footers" (signatures) from mailing lists. +.br +.IR NOTE : +This requires a directory with footer files to be given with option +.BR "\-L" =DIRECTORY. +.TP +.B "\-m" +"microsoft TOFU": +Hides TOFU as given by some microsoft mailers. (You all sure know these +fullquotes beginning with +.br +"----- Original Message -----" +.br +and some header lines...) +.TP +.BR "\-M, \-\-mua" =MUA +"mail user agent": +Turn on special treatment for some mail user agents. (Right now only +.BR mutt (1) +is supported, but more might be added in future.) +.TP +.BR "\-o" =FILE +"output file": +Define the file to be written *to*; the default output is to STDOUT. +.TP +.BR "\-p" =ADDRESS +"picky delivery:" +If we really find some TOFU, abort with exit code +.IR EX_UNAVAILABLE . +Otherwise redirect the message to ADDRESS. +.sp +Intended for use from within mail delivery agents (MDAs) or mail transport +agents (MTAs), so the message bounces if TOFU is detected, and does not get +on *your* nerves. :) +.sp +.B PLEASE be careful not to bounce messages to mailing lists! +.TP +.BR "\-P" =MESSAGE +"user defined bounce message for picky delivery": +You may specify your own bounce message to be returned when we try to deliver +an email and bounce it because there is TOFU inside. +.TP +.B "\-r" +"rip header off": +Hides all mail header lines. +.TP +.BR "\-S" [=n] +"supression of overlong signatures": +Signatures are to be n lines (not including the one containing dash-dash-space) +or less. If there are more, it is probably not that spirited after all. So +with this option you trade it for a +.B truely +nice line. +.br +If no n is given, default is 4. (\fINote\fR: We do not recommend using a +value other than 4. Consider this old-fashioned, but we actually do *like* +RFC conformance.) +.sp +.IR +Note : +The line containing "-- " ist not counted when testing for an overlong +signature, but it is included when displaying how many lines were deleted. +.TP +.B "\-s" +"signature deletion": +Hides signatures, i.e. all lines after a "signature dashes" line, +i.e. a line with three characters: dash-dash-space (no more, no less). +.TP +.B "\-t" +"TOFU deletion": +Hides "traditional style" TOFU, where each line begins with an +indent string like "> ". +.br +(You may edit the indent pattern in the script itself to suit your needs, +but it is surely +.I not +recommended at all.) +.TP +.B "\-w" +"whitespace deletion": +Hides trailing whitespace (sequences of space and tab). +CAVEAT: This may lead to interesting effects with crossposts +between mailing lists or with undetected signature attempts. +.TP +.B "\-v, \-\-version" +"version info": +Prints the current version number and release date, and exit. +.SH TOFU? +TOFU is an abbreviation which mixes German and English words; +it expands to "text oben, full-quote unten" which means +"text above - full quote below" and describes the style of so +many users who let their mailer or newsreader quote everything +of the previous message and just add some text at the top; +obviously they think that quoted text must not be changed at all. +This is quite annoying as it needlessly sends a lot of data +even when it is not required. Some editing of messages is desired. +Please point these people to the page +.I http://learn.to/edit_messages +- thank you! +.SH TROUBLESHOOTING +.TP +.IR Q : +I want to make my mailing list footer files match more different mailing +list footers. Can I use regular expressions, or how can I accomplish that? +.TP +.IR A : +Nope, regexp's do not work here. The comparison is made by the perl builtin +.IR index () +function (see +.B perldoc +for more detailed info), so you must exactly match the beginning of the +line. The longer the line you specify, the more precise the match; if +your line is empty you match unconditionally. +.PP +.TP +.IR Q : +I use the options -l and -L to supress mailing list footers when +displaying messages in +.BR mutt (1). +This does work sometimes, but sometimes it does not: the footer is not +detected, and therefore full quotes are not deleted and signatures are +detected as too long (which they aren't). +.TP +.IR A : +This might occur if the message is badly encoded, so mutt cannot resolve +all encoded characters, e.g. if you have an encoded message on a mailing +list, and majordomo appends a mailing list footer in a different encoding +(or even plain us-ascii). "-- " simply does not match "--=20". +.br +Another problem are non-us-ascii characters. Just avoid them, and +everything should work fine. +.br +See the preceding Q+A for a solution. +.PP +.TP +.IR Q : +I want to write a message which contains parts that should *not* be +deleted even when filtered with t-prot. Is this possible? +.TP +.IR A : +Yes, but please do not spread word of it. Make unobstrusive use of the +.I +verbatim +instruction: +.sp +#v+ +.br +This line is protected from being filtered by t-prot !!!!!!! +.br +#v- +.br +Text coming now is not. +.SH AUTHOR AND COPYRIGHT +Written by Jochen Striepe . +.br +Copyright \(co 2001, 2002 Jochen Striepe. +.PP +This is free software; you may use the source to whatever you like. +Just keep in mind to mention the original source of the script and the manpage +when you give it away. There is NO warranty; not even for MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. +.SH IDEAS AND INSPIRATION +Many good ideas, bug reports and support from Sven Guckes , +Gerfried Fuchs, Christian Borss, Bjoern Buerger, Bjoern Laessig, Martin +Neitzel and Ralf Doeblitz. +Many thanks to all of them! +.sp +Many thanks to Gerhard H. Wrodnigg who uses a TOFU protection script +in order to keep the responses to his cancel bot reasonably short. The +entire inspiration for this hack came from the "TOFU protection" line of +his script on many usenet postings. +.SH AVAILIBILITY +You can get the latest version from +.IR http://www.escape.de/users/tolot/mutt/ . +.SH BUGS +There is a problem when mutt gives a PGP verified or even a multipart +message to +.BR t-prot : +The information where the PGP encrypted/signed data or even attachments +begin and end is plainly embedded in the text, not really cleanly +recognizeable for +.BR t-prot . +The problem should be worked around by now, please send a bug report if +it does not work for you. +.PP +The script still does not solve the "Kammquoting" problem: Some bad MUAs +wrap long lines, so a zig-zag shaped text block is the result. This is +harder readable, and considered bad style. Your line length should not +exceed 80 characters - better even stick to 72-75 chars, so some place +is left for quoting. An anti "Kammquote" option is planned for one of the +next +.B t-prot +releases. +.PP +There probably are many more bugs in this piece of software. Feel free to +fix them - it's +.BR perl , +so you have the sources. Ah, and while you're at it, please drop a note to +the author - thanks in advance! +.SH "REPORTING BUGS" +Please send your bug report to . +.SH TODO +Fix bugs (see the +.I BUGS +section). Beside that, all main features should be implemented by now. +.SH "SEE ALSO" +.BR mutt (1), +.BR muttrc (5) +and the part about "display_filter", +.BR perl (1), +RFC822, +.I http://got.to/quote/ +(German language), +.I http://learn.to/edit_messages/ -- 2.39.2