1 .\" $Id: t-prot.1,v 1.192 2015/03/26 08:59:50 jochen Exp $
3 .TH T-PROT "1" "March 2015" "T-PROT"
5 t-prot \- TOFU Protection - Display Filter for RFC 5322 messages
7 .BI "t\-prot [" OPTIONS "]..."
10 This program is a filter to improve the readability of internet
11 messages (emails and usenet posts) by *hiding* some annoying parts,
12 e.g. mailing list footers, signatures, and TOFU (see definition below),
13 as well as squeezing sequences of blank lines or punctuation.
14 The program also detects TOFU or a high quoting ratio in a message (so
15 you may take appropriate action, e.g. when submitting messages to a mailing
16 list or a news server).
18 The filter is written in Perl and relies on input to be a single
19 message conforming to RFC 822 or its successors, RFC 2822 and RFC 5322.
20 In messages conforming to MIME (RFCs 2045-2049) t\-prot handles text/plain
21 parts, others are not touched.
23 Already reformatted messages are handled well: the script was
24 initially designed to cope with the output of the MUA mutt (which
25 is the reason for not using standard CPAN modules for handling
28 T\-prot offers example configuration files for mutt and its fork mutt\-kz,
29 Heirloom mailx and metamail. Also coming with the t-prot package is the
30 example S-Lang macro t\-prot.sl for using t-prot from within slrn. There
31 is a proof\-of\-concept filter for INN2, which you will have to adapt
32 to the needs of the news site you host. For use with sendmail's
33 alias(5) file, please see below (the option
34 .BR "\-p" " provides an example line)."
37 If you do not specify any options, t\-prot does ... nothing. Every
38 feature you want must be turned on explicitly.
39 Admittedly, we have quite a number of options for t\-prot. To limit
40 confusion they are grouped into five sections: Input/Output Options,
41 Advertisement And Mailing List Footers, Filtering Options, Detection
42 Options, and Other Options. While the others should be quite clear,
43 filtering and detection might deserve a word (or two).
45 If you want to tune the appearance of your mail from within your MUA
46 (or news messages from within your NUA), then go for the Filtering
49 If you want to use t\-prot to check on mails before they are submitted
50 to mailing lists, fed to your news server, or delivered by your MDA,
51 then have a peek at the Detection Options section. You may accept or
52 reject/bounce messages depending on t\-prot's result.
53 .SH INPUT/OUTPUT OPTIONS
56 Defines an input file; default is '\-' i.e. STDIN.
59 Defines the output file; default is STDOUT.
62 Input consists just of the message's body. There are no RFC 5322 header
66 This does not work with \-\-pgp\-short, and multipart messages will not
67 be detected due to missing headers.
69 .B "\-\-lax\-security"
70 Allow insecure writing method. DO NOT USE UNLESS YOU REALLY KNOW WHAT
71 YOU ARE DOING. (This ugly workaround is needed for some early mutt versions
72 and should NEVER be used as a default, otherwise it will probably turn into
75 You can use this option safely to enable
77 /dev/null (or other files which cannot be changed with the user's
80 .BR "\-\-max\-lines" =x
81 Maximum number of lines a message may count (with headers). If the message
82 is longer than x lines, the message will not be processed but printed
83 unmodified. Exit status will be EX_DATAERR except when called with \-Mmutt
85 .SH ADVERTISEMENT AND MAILING LIST FOOTERS
88 "commercial signature":
89 Hides "footers" (signatures) from commercial email providers.
91 This option compares the last lines of the message body with any
92 footer file found in the directory specified with
93 .BR "\-A" " DIRECTORY"
94 (which is mandatory for this option). The comparison is done by perl's
101 This option is not needed if
106 "enable aggressive ad footer matching":
107 With this option enabled, t-prot makes footer detection really greedy: We
108 assume that commercial email providers aren't even frightened to append
109 changing texts *under* their ads which are appended to the message body.
110 Because these texts even have changing lengths we simply detect the
111 lines of the footer *anywhere* in the body of the message and assume that
112 everything below belongs to the footer. (Man, if life where always that
115 With this option even GMX ads should be easy to hide -- you buy this with
116 a slight performance hit (which is the reason this option is disabled by
117 default), and with the possibility that sometimes the algorithm is just a
121 This requires a directory with footer files to be given with option
122 .BR "\-A" " DIRECTORY".
124 .BR "\-A" " DIRECTORY"
125 "ad footer directory":
126 Defines the directory which contains the advertisement list footers (one
127 footer per file) which are to be tested when removing them with options
132 This option is also needed if you do not want signature lengths to be
133 counted wrong or fullquotes get undetected when an ad footer is
134 appended at the bottom of the message (especially when using
141 Hides "footers" (signatures) from mailing lists. Footer detection works like the
146 This requires a directory with footer files to be given with option
147 .BR "\-L" " DIRECTORY".
154 "enable aggressive mailing list footer matching":
155 With this option enabled t-prot makes footer detection really greedy: Should
156 be helpful with broken list servers, or even if your email provider munges
157 the bodies of your messages.
159 Works similar to \-\-ftr\-ad, just that it is intended for mailing list footers.
162 This requires a directory with footer files to be given with option
163 .BR "\-L" " DIRECTORY".
165 .BR "\-L" " DIRECTORY"
166 "list footer directory":
167 Defines the directory which contains the mailing list footers (one footer
168 per file) which are to be tested when removing them with the options
173 This option is also needed if you do not want signature lengths to be
174 counted wrong or fullquotes get undetected when a mailing list footer is
175 appended at the bottom of the message (especially when using
181 .BR "\-\-bigq" [=n[,x]]
183 Blocks of quotes with more than n lines will be shrunk to x lines.
184 Defaults are 30 for n and 10 for x.
188 Squeezes a sequence of blank lines to just n blank lines. n defaults to 2.
191 Tolerate unified diff (see
195 appended after the signature (which usually makes the signature too long
198 Also, protect diff standard output from hiding (which would otherwise be easy
204 Squeezes a sequence of four or more dots, exclamation marks, or question marks
205 to only three dots or marks, respectively.
208 Fix broken quotes to adhere to RFC 3676 by removing spaces between quote
209 characters and adding a space after them.
212 This may produce false positives if spaces in between quote characters
213 are intended (thus changing the quoting level, see RFC 3676 for details).
216 Hides TOFU as produced by Novell Groupwise.
220 Tries (not too aggressively) to fix those broken zig-zag-shaped lines
221 wrapped around by some MUAs which are known as "Kammquoting" in German.
224 This option is considered stable by now. However, sometimes Kammquotes
225 should have been removed but weren't. Please send a bug report if this
226 happens to you (after carefully reading the BUGS and REPORTING BUGS
227 section of this man page, that is).
229 Please also note that enabling this option is quite a performance hit.
232 Minimum length difference between two lines for wrapped line detection on
233 Kammquotes. For details, please see the source code.
235 Anyway, lower values make the algorithm more aggressive, higher values
236 make Kammquotes harder to detect. Default is 20.
242 Maximum line length for wrapped line detection on Kammquotes. For
243 details, please see the source code.
245 Anyway, higher values make the algorithm more aggressive, lower values
246 make Kammquotes harder to detect. Default is 80.
252 Minimum line length for wrapped line detection on Kammquotes. For
253 details, please see the source code.
255 Anyway, lower values make the algorithm more aggressive, higher values
256 make Kammquotes harder to detect. Default is 65.
261 .BR "\-\-locale" =LOCALE
262 Specify which locale to use for correct parsing of your MUA's formatting
263 of the displayed message (usually it is the locale your MUA uses). Right
264 now this option is only used when
267 is specified, but this may change in future. You need the Perl module
277 with locales, t-prot will only work correctly if you specify the corresponding
278 locale string. Alternatively, you can use the environment variables
283 to specify the locale string.
286 You also have to make sure you are running t-prot
292 versions. T-prot detects
297 locales of the recent stable versions of those programs, earlier versions
298 might not work well with a recent version of t-prot.
300 .BR "\-M, \-\-mua" MUA
302 Turn on special treatment for some mail user agents. (Right now only
306 are supported, but more might be added in future.)
308 If your MUA is supported by this feature you must ensure t-prot makes
309 use of it when called from within your MUA to work as desired.
313 Hides TOFU as given by some Microsoft mailers. (You all surely know these
314 fullquotes beginning with
316 "\-\-\-\-\- Original Message \-\-\-\-\-"
318 and some header lines...)
321 Burn CPU cycles trying to be smart with MS style TOFU: If there are PGP
322 signed parts inside the TOFU, the text still might conceal other message
323 parts and therefore should not be deleted.
325 Please note that this is probably just a waste of time because most
326 MS Outlook users who do produce this kind of TOFU won't care about
327 making their messages the least bit readable or even predictable. So
328 this option will probably just be interesting for mutt message hooks
329 (to activate it on demand when you
330 .BR know " the sender tries to write legible messages)."
334 .BR "\-Mmutt\-kz" " and"
338 Move PGP and SSL verification output to bottom; requires
342 .B \-\-pgp\-move\-vrf
343 Move PGP and SSL verification output to bottom only if verification shows a
344 good signature and the signature could be verified as authentic (using a
345 trust path). If there is any problem with the signature, the PGP output
346 should not be moved so the user is more likely to notice. Requires
351 If gpg is terminated before finished (e.g. hitting Ctrl-C, or using
353 we cannot always detect if the check was interrupted. Though t-prot tries
354 to be smart, there will be false positives.
357 Hide non-relevant PGP key uids; requires
363 Hides all mail header lines.
366 Subject lines with multiple reply prefixes (Re: and translations into other
367 languages) get squeezed to only one prefix.
370 "suppression of overlong signatures":
371 Signatures are to be n lines (not including the one containing dash-dash-space)
372 or less. If there are more, it is probably not that spirited after all.
373 So with this option you trade it for a
377 If no n is given, default is 4. (We do not recommend using a value other
378 than 4. Consider this old-fashioned, but we actually do *like* RFC
382 The line containing "\-\- " ist not counted when testing for an overlong
383 signature, but it is included when displaying how many lines were deleted.
386 "signature deletion":
387 Hides signatures, i.e. all lines after a "signature dashes" line,
388 i.e. a line with three characters: dash-dash-space (no more, no less).
391 Sanitize headers "To:", "From:" and "Subject:": Quoted-printable gets
392 fixed to the corresponding chars. German Umlauts are translated to their
393 "ae", "oe", "ue" pendants.
395 Useful e.g. for searching by subject within MUAs like Berkeley mailx.
397 .BR "\-\-sigsmax" [=n]
398 "maximum number of tolerated signatures":
399 Here you can define how many signatures you accept to be treated as such.
400 (Most significant behaviour is when microsoft style quotes are removed.
401 Experts please see the code for the more subtle implications of this
404 Leave empty or specify zero to have an unlimited number of sigs.
408 "SpamAssassin workaround":
409 SpamAssassin (available at
410 .IR http://spamassassin.org/ ") often is configured that it adds some"
411 lines to the message body containing information about the spam criteria
412 which were found matching for the message. This option enables an extra
413 test to avoid false positives for Microsoft style TOFU on such messages.
417 Hides "traditional style" TOFU, where each line begins with the
421 "whitespace deletion":
422 Hides trailing whitespace (sequences of space and tab).
423 CAVEAT: This may lead to interesting effects with crossposts
424 between mailing lists or with undetected signature attempts.
425 .SH DETECTION OPTIONS
428 "user defined bounce message for picky delivery":
429 You may specify your own bounce message to be returned when we try to deliver
430 an email and bounce it because there is TOFU inside. See
433 .BR "\-p" " [ADDRESS]"
435 If we really find some TOFU, abort with exit code
437 Otherwise redirect the message to ADDRESS if given.
439 Intended for use from within mail delivery agents (MDAs) or mail transport
440 agents (MTAs), or even from within INN, so the message bounces if TOFU is
441 detected, and does not get on *your* nerves. :)
443 As an example for usage with
445 put this line into your alias file and invoke
448 notofu: |"/usr/local/bin/t\-prot \-mt \-p=user@mydomain"
450 This will bounce messages for <notofu@domainname> if any TOFU is detected
451 inside the message, and deliver it to <user@mydomain> otherwise.
453 that TOFU is only detected if you specify
458 .B PLEASE be careful not to bounce messages to mailing lists!
460 .BR "\-\-check" [=FLAGS]
461 Run checks. If successful, print an error message and quit with
462 an appropriate exit code. Useful e.g. for rejecting messages from
465 Flags are separated by commas (no whitespaces), and can be the
466 following (right now just one flag):
470 If the quoting ratio is n or more, the message is rejected. Must be
471 between 0 and 1, or else it is entirely disabled. Default is 0.75
472 (i.e., 75% of the message lines are quotes).
475 Print envelope info to syslog when bouncing TOFU contaminated email.
476 Default syslog facility is mail.debug. Requires
481 Displays a short help text with a summary on all options, and exits.
483 .B "\-v, \-\-version"
484 Prints the current version number and exits.
486 The environment variables
491 are read and respected when interpreting output by mutt / mutt\-kz or gnupg
492 (unless they are overruled by the \-\-locale option). T-prot's own
493 output is English regardless of any locale setting.
495 On program exit, t-prot uses exit codes from
496 .I /usr/include/sysexits.h
497 and thus behaves in a manner that sendmail and others understand when
500 Currently, the codes used are
518 If, however, perl fails to compile and execute t-prot, perl's normal
519 exit codes will be returned.
521 TOFU is an abbreviation which mixes German and English words;
522 it expands to "text oben, full-quote unten" which means
523 "text above - full quote below" and describes the style of so
524 many users who let their mailer or newsreader quote everything
525 of the previous message and just add some text at the top;
526 obviously they think that quoted text must not be changed at all.
527 This is quite annoying as it needlessly sends a lot of data
528 even when it is not required. Some editing of messages is desired.
529 Please point these people to the page
530 .I http://www.river.com/users/share/etiquette/edit.html
533 There are several ways to fine-tune t-prot's performance:
535 Some command line options are quite grave a performance hit -- do not
536 use \-k and especially \-\-ms\-smart if you are content without them.
538 Checking for special footers is very costly as well. Put as few footer
539 files as absolutely needed in any footer directory.
541 All PGP related options are eating up lots of CPU time. Try to avoid them
542 on unsigned and unencrypted messages.
544 When calling t-prot from within mutt (or mutt\-kz), you might use mutt's
545 folder-hook and message-hook facilities to turn options on only when
546 needed, e.g. to set up a different footer directory for each mailing list
551 I want to make my mailing list footer files match more different mailing
552 list footers. Can I use regular expressions, or how can I accomplish
556 Nope, regexp's do not work here. The comparison is made by the perl
561 for more detailed info), so you must exactly match the beginning of the
562 line. The longer the line you specify, the more precise the match; if
563 your line is empty you match unconditionally.
567 I use the options \-l and \-L to suppress mailing list footers when
568 displaying messages in
570 This does work sometimes, but sometimes it does not: the footer is not
571 detected, and therefore full quotes are not deleted and signatures are
572 detected as too long (which they aren't).
575 This might occur if the message is badly encoded, so mutt cannot resolve
576 all encoded characters, e.g. if you have an encoded message on a mailing
577 list, and majordomo appends a mailing list footer in a different encoding
578 (or even plain us-ascii). "\-\- " simply does not match "\-\-=20".
580 Another problem are non-us-ascii characters. Just avoid them, and
581 everything should work fine.
583 See the preceding Q+A for a solution.
587 I want to write a message which contains parts that should *not* be
588 deleted even when filtered with t-prot. Is this possible?
591 Yes, but please do not spread word of it. Make unobstrusive use of the
598 This line is protected from being filtered by t\-prot !!!!!!!
602 Text coming now is not.
604 Written by Jochen Striepe <t\-prot@tolot.escape.de>.
606 All of the documentation and software included in the t-prot releases
607 is copyrighted by Jochen Striepe (except when explicitly stated otherwise).
609 Copyright \(co 2001-2015 Jochen Striepe. All rights reserved.
611 Redistribution and use, with or without modification, are permitted
612 provided that the following conditions are met:
614 1. Redistributions of source code must retain the above copyright notice,
615 this list of conditions and the following disclaimer.
617 2. All advertising materials mentioning features or use of this software
618 must display the following acknowledgement:
620 This product includes software developed by Jochen Striepe and others.
622 3. Neither the name of the author nor the names of any contributors may
623 be used to endorse or promote products derived from this software without
624 specific prior written permission.
626 THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND ANY
627 EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
628 WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
629 DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR ANY
630 DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
631 (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
632 LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
633 ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
634 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
635 SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
636 .SH IDEAS AND INSPIRATION
637 Many good ideas, bug reports and support from (in alphabetical order) Bjoern
638 Buerger, Bjoern Laessig, Christian Borss, Gerfried Fuchs, Martin Neitzel, Martin
639 Dietze, Matthias Kilian, Ralf Doeblitz, Sven Guckes and many more (see the
640 ChangeLog for active contributors). Many thanks to all of them!
642 Many thanks to Gerhard H. Wrodnigg who uses a TOFU protection script
643 in order to keep the responses to his cancel bot reasonably short. The
644 entire inspiration for this hack came from the "TOFU protection" line of
645 his script on many usenet postings.
647 You can get the latest version from
648 .IR http://www.escape.de/users/tolot/mutt/ .
650 There is a problem when mutt gives a PGP verified or even a multipart
651 message to t-prot: The information where the PGP encrypted/signed data
652 or even attachments begin and end is plainly embedded in the text, not
653 really cleanly recognizable for t-prot. The problem should be worked
654 around by now, please send a bug report if it does not work for you.
656 Please note that t-prot development happens on
657 .BR "current stable perl versions only" .
658 If you do run t-prot on earlier (or unstable) perl versions, you might
659 encounter perl compiler bugs (or funny t-prot behaviour). One solution
660 is to upgrade your perl, another is simply to write a bug report. If
661 you do not run a current perl version, please include this information
668 * you found it in the TODO file coming with the distribution. We do
669 know those and try to fix them as soon as possible.
671 * you have an old t-prot version. If you encounter a problem, first
672 see if there is a new t-prot version which fixes the issue. If you
673 upgraded to the latest version and it still occurs, a bug report is
676 If you noticed a bug when processing a message and want to provide the
677 t-prot team with some useful info, please:
679 * if invoking t-prot by mutt's
681 facility, just set display_filter to something like
683 "tee ~/foobar | t\-prot <your options>"
685 and include ~/foobar in the bug report -- this way we might reproduce
686 the bug much easier if you are using a different environment than we do.
688 * provide information on what command line options you use t-prot
689 with, what perl version t-prot runs on your system, and what else might
690 be important to enable us reproducing the bug.
692 Send your bug report to
693 .IR <t\-prot\-bugs@tolot.escape.de> .
698 section). Beside that, all main features should be implemented by now.
699 See the TODO file for more information.
704 and the part about "display_filter",
708 RFCs 2045-2049, 3676 and 5322,
710 .I http://freshmeat.net/articles/t\-prot/
711 (a nice, solid introduction),
713 .I http://got.to/quote/
716 .I http://www.river.com/users/share/etiquette/edit.html
717 (the Learn To Edit Messages HowTo has found a new home).
projects / pkg / t-prot.git / blob