Animail

Table of Contents

1. Command line options
2. The filtering system
2.1. Using Animail with other filters (SpamAssassin, etc)
3. Animailrc configuration reference
3.1. Server options
3.2. Global options
3.3. Post-Filter options
4. Tricks

1. Command line options

--help or -H

Show help

-o [arrival | small | bug] or --download-order=[arrival | small | big]

-f [file] or --mbox-file=[file]

-v or --verbose

Verbose mode

-s [size] or --max-size=[size]

Maximum size (bytes) of messages to be downloaded

-n [max] or --max-to-download=[max]

Maximun number of messages to download

-S or --deliver-smtp

Resend the messages to the local SMTP

-B or --syslog-output

Write all the output to the syslog

-t or --quiet

Quiet mode

-F or --only-filter

Don't download, only filter

-l or --no-summary

Don't show the summary

-C or --check

Check for new mail, and print how many mails we have in each remote mailbox but don't download them. The output is very easy to parse by other programs. The program also returns 1 if there is new mail and 0 in the opposite.

-M or --fetchmail

The same that -C but with a fetchmail-compatible output.

-O or --only-from=

Only download messages from the given comma separated list of servers on this run. The servers parameter is a server alias NOT a server address (so you must configure an 'Alias:' command in the animailrc for each server you will eventually want to use with this).


2. The filtering system

The filtering system relies on three elements:
  1. The accept file
  2. The deny file
  3. The postergated messages
The accept file is (like the deny file) a file in which each line is a regular expression (man regexp). If the header of the message currently being downloaded match one of those regular expressions, the message is automatically accepted, without having to check the deny file. Those expressions Those regular expressions will usually be the ones matching the messages from out friends, work partnerts, mailing list we're subscribed to, etc.

The path to the accept file is specified in the animailrc file with:
AcceptFile: /path/to/acceptfile

Just including the above command in the animailrc activates his use.

The deny file has the same format that the accept file, but insted of making the messages matching one of their regular expressions to be accepted has the reverse effect: They are deleted from the server without even downloading his body. The regular expressions we write in this file will usually match the email addresses of known spammers. Of course, using regular expression syntax we can match anything contained in the header, not only the 'From' address but also the number of people in the Cc: field and almost any other thing we can think of (regular expressions are extremelly powerfull).

The path to the deny file is specified in the animailrc file with:
Filterfile: /path/to/filterfile

Just including the above command in the animailrc activated his use.

So depending of we using one, both or none of the above files we're going to have one or other filtering system. If we only define the deny file (Animail only allowed this until version 1.1.28) we are going to be using a 'black list' scheme. This scheme worked very well in 1998 when Animail started to be developed, because there weren't soo many spammers and most of the spam come from a few addresses. Unfortunately nowadays this filtering system is not enough because there are a lot of spammers and they change frecuently his address. A solution to this oculd be to try some very generic regular expressions to filter subsentences like 'lot of money' o 'life for ever' in the subject, but unfortunately this can result in legitimate email being deleted (for example, a friend sending us an email with the subject "I found work for you which can make you earn lots of money".

Trying to a find a solution for this, Animail (version 1.1.28 and up) allows another two filtering systems (inspired by the idea the software TMDA is using but applied to a different level. TMDA is distributed with a BSD license, but Animail doesn't share any code with it.):

  1. To deny anything not in the accept file.
  2. To put messages not in the accept file or in the filter file in a 'postergated' state and send a polite message back asking for a confirmation

The first option is very aggresive, but can be interesting for people that only want to receive email from limited sources (like mailing list and some friends). To use this option besides having specified the path to the accept file we need to add to the global section of the animailrc:
NotInAcceptFile: Delete

The second option, undoubtedly more usefull for most people, is that each time we receive and email from a source not in the accept or deny file that message is put 'on hold' in the animailDelayed directory under the .animail directory of out home dir, and a polite reply is sent specifiying why is message has not been delivered yet (anti spam measures) and asking to reply to that autoreply so his address is added to the accept file. When Animail receive that reply all the messages 'on hold' are delivered, and the original address is added to the accept file (we can configure Animail not to add automatically the address, see the global options reference.)

Is important to note that is we're using this filtering system we've to add the different mailing list address we're subscribed to the accept file, and some other strings matching some automatically generated messages we want to receive (like "MAILER-DAEMON", "subscri", etc).

Anyway when the operation summary is shown, we can see what email address have been put on hold and looking for some days those address we can figure what strings to put on the accept file (we can manually read the messages put on hold opening the mbox files under .animail/animailDelayed with our favourite mail reader.)

2.1 Combining Animail with other filters (SpamAssassin, procmail, etc)


As the tittle of this section suggests, Animail can be used with other external filters to combine his filtering skills with the ones already on Animail. Those filters (you can specify any number of them) will be called just after downloading a message and just before delivering it; depending of the return value of some output of that filter the message will be delivered, ignored, or copied to a log mailbox (so you can see if those filters are not deleting important mail). All the commands discussed on the examples of this subsection are detailed on the <PostFilter> command block reference.

Doing it is very easy: You just have to create a new PostFilter block and configure in a way suitable for the program you are using. You can also add options to copy the filtered messages to a log mailbox, to copy the output of the post-filter instead of the original message to that mailbox (this is useful for programs which output is the original message modified, like SpamAssassin). It is more easy to see with a few examples:

PostFilter block for SpamAssassin:
<PostFilter>
PostFilterName: SpamAssassin
Path: /usr/bin/spamassassin
KillerProgramOutput: ^X-Spam-Flag: YES.*$
SavePostFilteredMails: stdout
<PostFilter>

The first command, after the tag, tells the name of the filter (for the program output). The 'Path' command is the path to the filter executable and third is a regular expression that if match with the program output will filter the message. The last command tells Animail to save the standart output of spamassasin in a special mailbox ('spambox'). That's all. Well, not exactly; you also have to put this on the global section:

PostFilteredMailsMbox: /home/yourhome/spambox

This command specifies the spambox to use to save filtered mails.

More examples:

PostFilter block for spamc (SpamAssassin C client):
<PostFilter>
PostFilterName: SpamAssassin-spamc
Path: /usr/bin/spamc
KillerReturnValues: 1
SavePostFilteredMails: stdout
<PostFilter>

PostFilter block for Bogofilter:
<PostFilter>
InputMessageByStdin: Yes PostFilterName: Bogofilter
Path: /usr/bin/bogofilter
KillerReturnValues: 0
SavePostFilteredMails: no
<PostFilter>

How to add a new postfilter block for other commands: The first thing you have to see is if that program read the mails from his standart input or are specified as a file to his command line, maybe after some parameter. In the first case you don't have to specify nothing to Animail; stdin input is the default. In the second case, you must write inside the postfilter section: InputMessageByStdin: No and in the Options command you have to put a '%M' control code where you would usally write the path to the message (along with other possible options). Now you have to look how the filter reports when a program is spam or not; Animail can handle to cases: Where the program returns some special return code if the message if spam (like 'Bogofilter' returning 0) or if the program output by his standard output or standard error some text when it finds a spam message (this is the case with SpamAssassin, which outputs X-Spam-Flag: YES). In the first case the command to use is KillerReturnValues:, which parameter is a comma separated list (that could contain a single element) of numeric return codes, and in the second case is KillerProgramOutput followed by one regular expression that have to match the program output that reports spam. Maybe a program can report spam in both ways (like SpamAssassin spamc client); in this case you're safe writing both commands on the config file (they are not exclusive). Finally you have to decide if you want to see the filter output after every run (if there is some output, that is) or you want to hide that output (that will still be parsed by Animail if KillerProgramOutput is specified). This is done with the ShowFilterOutput command that can have the values yes/no, defaulting to 'no'.

Finally you have to decide if you want the filtered messages to be copied to another mailbox (I call it 'spambox') where you could check later for false positives (not spam tagged as spam). This is done with the SavePostFilteredMails command that can have the values 'yes' for coying the original mail to the spambox, no for don't, and 'output' to copy the filter program output instead of the original mail (this is useful with filters like SpamAssassin that print on his standard output a new version of the email of his standard input). If you set this command to anything besides 'no' you will also have to set the globa section command PostFilteredMailsMBox to a writable path, or Animail will issue a warning every time a spam-message is to be copied.

3. Animailrc configuration reference

Note: This file must exist in each Animail user home directory under the directory .animail/ , there is not a system-wide configuration file.

This file is composed of one or more server groups and optionally postfilter groups and a global group. Each server groups defines the options for a specified server and the global option group defines... well, the global options. The postfilter groups specify the options for using an external programs as a help for the filtering system.

Servers groups are delimited between the starting tag <Server> and the ending tag </Server> , and global the global options groups by <Global> and </Global> , so the general structure for an animailrc file could be:

<Server>

[Server commands here]

</Server>


<Server>

[Another server commands here]

</Server>

... [More Server options group if you need to download mail from more servers]

<PostFilter>

[Postfilter commands here]

</PostFilter>

<PostFilter>

[Another postfilter commands here]

</PostFilter>

<Global>

[Global options here]

</Global>

The commands of each groups must be separated from threir parameters by a color. All commands and boolean (yes/no) parameters are case insensitive.


3. Options Reference


3.1 <Server> options

1. Address:
Parameter: String / IP Address in point notation
Default value: localhost
Internet address of the incoming mail server to use.

2. Protocol:
Parameter: Protocol string (POP3, IMAP4, APOP)
Default value: POP3

Protocol to use. Currently supported protocols are POP3, APOP and IMAP4.

3. SSL:
Parámeter: Boolean (yes/no)
Default value: no

This options tells Animail to use the Secure Socket Layer protocol for the transfers if the remote server supports it. Currently only works with POP3.

4. Port:
Parameter: Numeric
Default value: 110 for POP3, 995 for POP3+SSL, 143 for IMAP4
Port to use with the mail server.

5. UserName:
Parámeter: String
Default value: None
Username to use with the mail server account.

6. Password:
Parameter: String
Default value: None
Password to use with the mail server account.

7. KeepOnServer:
Parámeter: Boolean (yes/no)
Default value: no
Specify if the messages must be deleted of the mailserver after the download or not. This is useful to experiment without the risk of losing messages.

8. ResendTo:
Parameter: String list, comma separated of usernames or email addresses
Default value: Local user running Animail
Tells Animail to forward all downloaded messages to the given user or email address. If this option is not activated messages will be forwarded to the user running Animail. The parameter can be a single address or a comma separated list of addresses.
Note: Depending of the local mail server configuration, it's posible that the domain must be specified even with local users (adding a @localhost suffix after the username.)

9. Deliver_mbox:
Parameter: String (path to file)
Default value: $HOME/mbox
This option overrides the globally configured delivery option for this server. Activate delivery of this server messages to a Mbox-style mailbox (standar Unix format which can be readed with most conventional Unix mail clients), and specifies the path to the mbox file.

10. Deliver_maildir:
Parameter: String (path to a directory)
Default value: $HOME/Maildir/
This option overrides the globally configured delivery option for this server. Activate delivery of this server messages to a Maildir mailbox (as used by QMail) and specifies the path to the Maildir directory.

11. Deliver_localmta:
Parameter: Boolean (yes/no)
Default value: yes
This option overrides the globally configured delivery option for this server. Activate delivery of this server messages to a local MTA (localhost, port 25) using the SMTP protocol. This is the recommended (and default) way of operation.

12. Deliver_pipe:
Parameter: String (command line)
Default value: /usr/sbin/sendmail -bm

This option overrides the globally configured delivery option for this server.

Activate delivery of this server messages to the command specified with the parameter. The default parameter value is /usr/bin/sendmail -bm.
If this option is found with a value Animail will use the command specified to deliver downloaded messages instead of using the SMTP protocol or any of the other delivery methods (currently mbox and maildir). The parameter can be any command that accepts RFC822 messages on his standart input. The address of the destination of each email will be appended to the command before opening the pipe so if the command needs an special parameter to accept the destination address you need to pass it (for example, a line for procmail could be procmail [parámeters] -f).


3.2 <Global> options

1. MaxSize:
Parameter: Numeric (in bytes)
Default value: Unlimited
This command specifies the maximum size of a message to be downloaded. Those messages that won't be downloaded will be deleted on the server.

2. MaxNumber:
Parameter: Numeric (number of emails)
Default value: Unlimited

Limits the number of messages to download in a session. Those messages that won't be downloaded will be left on the server. If this option is omited there is no limit to the number of messages to download in a session.

3. Deliver_mbox:
Parameter: String (path to file)
Default value: $HOME/mbox
Activate delivery of messages to a Mbox-style mailbox (standar Unix format which can be readed with most conventional Unix mail clients), and specifies the path to the mbox file.
This option can be overriden for some servers specifiying different delivery options inside that servers options block.

4. Deliver_maildir:
Parameter: String (path to a directory)
Default value: $HOME/Maildir/
Activate delivery of messages to a Maildir mailbox (as used by QMail) and specifies the path to the Maildir directory.
This option can be overriden for some servers specifiying different delivery options inside that servers options block.

5. DownloadOrder:
Parameter: String (arrival, firsbig of firstsmall)
Default value: arrival
Specify the messages download order based on his size. It can have three parameters:


6. Deliver_localmta:

Parameter: Boolean (yes/no)
Default value: yes
Activate delivery of messages to a local MTA (localhost, port 25) using the SMTP protocol. This is the recommended (and default) way of operation.
This option can be overriden for some servers specifiying different delivery options inside that servers options block.

7. Deliver_pipe:
Parameter: String (command line)
Default value: /usr/sbin/sendmail -bm

Activate delivery of this server messages to the command specified with the parameter. The default parameter value is /usr/bin/sendmail -bm.
If this option is found with a value Animail will use the command specified to deliver downloaded messages instead of using the SMTP protocol or any of the other delivery methods (currently mbox and maildir). The parameter can be any command that accepts RFC822 messages on his standart input. The address of the destination of each email will be appended to the command before opening the pipe so if the command needs an special parameter to accept the destination address you need to pass it (for example, a line for procmail could be procmail [parámeters] -f).
This option can be overriden for some servers specifiying different delivery options inside that servers options block.

8. FilterFile:
Parameter: Path to file
Default value: none

Using this option you tell Animail to turn on regular expression filtering. Regular expressions must be written in the file specified in the path, one regular expression in each line.

9. AcceptFile:
Parameter: Path to file
Default value: none

Using this option you tell Animail to auto accept the messages matching the regular expressions in that file. That regular expressions must be written one in each line. (See Filtering.)

10. NotInAcceptFile:
Parameter: string ('none', 'reply', 'ignore' or 'delete')
Default value: 'none'

Specifies with action to do when a message that is not in the accept or in the filter file is received. If the parameter is 'delete' the file will be deleted from the server and if it is 'reply' a reply will be sent to the sender asking for a confirmation (see Filtering.). If the value is 'none' the message will be normally delivered and if it is 'ignore' it will not be delivered, but it will also don't be deleted from the server.

11. ReplyAddress:
Parameter: string (email address)
Default value: nothing (it must be specified)

Address Animail will use as the Reply-To: of the auto messages to send.

12. ReplySubject:
Parameter: string (email subject)
Default value: 'Auto-generated message'

Subject to be used in the auto-replys to messages not in the accept or in the deny files. Animail will add the address specified in the ReplyAddress command to the end of that subject string so it can't get confused by others animails' messages.

13. ConfirmSubject:
Parameter: string (email subject)
Default value: 'Ok, your messages have been delivered'

Subject to be used in the replys to email confirmations.

14. FileWithReply:
Parameter: string (path to file
Default value: None (in this case the default message will be the one that can be read in the file general.py of the Animail distribution)

File with the body of the message to be sent in the replies to emails not in the accept or in the filter file asking the user to a confirmation just replying to this message, so his email can be finally delivered and his address added to the accept file. If this command is not specified a polite message in several languages (currently english, spanish and italian, looking for help with more translations) will be used.

15. FileWithConfirmation:
Parameter: string (path to file)
Default value: None (in this case the default message will be the one that can be read in the file general.py of the Animail distribution)

File with the body of the message to be sent in the replies to users confirmations. If this command is not specified a polite message in several languages (currently english, spanish and italian, looking for help with more translations) will be used.

16. AutoAcceptConfirmations:
Parameter: Boolean (yes/no)
Default value: Yes

This option specifies if after a confirmation to an Animail request for confirmation the sender must be automatically added to the accept file.

17. OnlyOneReply:
Parameter: Boolean (yes/no)
Default value: No

Animail keeps a list of replied addresses in the file $HOME/.animail/repliedList.dat (DON'T EDIT THAT FILE!!). This option makes Animail only send a reply to the first message coming from an address not matching a regular expression in the accept or in the filter file (the operation summary will continue to show the postergated messages).

To prevent excesive startup time and memory usage, the list of replied messages has a maximum size of 500 entries (older entries will be removed when that sized is exceeded).

18. Timeout:
Parameter: Numeric (time in seconds)
Default value: 60
Limit how many seconds the program can wait to receive data from a socket (the time the connection with a server is halted). After this timeout has expired the connection is closed and Animail continues with the next server. The default value for this parameter is to issue a timeout after 60 seconds, use 'Timeout:0' to disable using a timeout on sockets.

19. Mailer:

[deprecated]
Parameter: String (command)
Default value: None
Mailer to use for reading messages (clicking on the "Read" button in the graphical summary window).

20. Colorize:
Parameter: Boolean (yes/no)
Default value: yes
Apply nice ANSI colors to the console output.

21. Reconnect:
Parameter: Boolean (yes/no)
Default value: yes
When some server fail (for whatever reason) during the operation, Animail will try to reconnect to that server at the end of the session to delete already downloaded messages so the next time we try to download messages from that servers you wont have duplicated messages on you mail spool. Enabled by default.

22. WaitBeforeReconnection:
Parameter: Numeric (time in seconds)
Default value: 30
Time before reconnecting to a failed server (see the Reconnect option). This time start counting in the moment that the server failed. This is necessary because some servers (usually POP3 servers) don't allow the clients to do more than one connection at a time, and the server needs some time to know that the connection is lost and close that session.

23. PostFilteredMailsMBox:
Parameter: String (Path to mailbox file)
Default value: None
This options tell Animail to save the message filtered by the post-filters (see the Using Animail with other filters and Post-Filter options sections) on a mailbox ('spambox') file. For this option to be useful you must also set the 'SavePostFilteredMails' option of one or more <PostFilter> command blocks to 'Yes', 'Output' or 'Stdout'.

24. SMTPSpamCodes:
Parameter: Comma separated list of three-digits SMTP codes
Default value: None
Using this option will instruct Animail to delete from the server they were downloaded the messages to which the local SMTP server issue an SMTP error code matching one of the codes on this option list, insted of leaving the offending message in the server as would happen with other error codes. This is usually used to enable the local SMTP anti-spam features, thus the option name.


3.3 <PostFilter> options

1. PostFilterName:
Parameter: String
Default value: None (must be specified)
Name for the filter. It will be using whenever Animail needs to name this filter in his output.

2. Path:
Parameter: String (file path)
Default value: None (must be specified)
This is the path to the filter program. It must be a file, and that file must have permissions to allow Animail reading and executing it.

3. InputMessageStdin:
Parameter: Boolean (Yes/No)
Default value: Yes
This switch tells Animail is the email must be feed to the postfilter program using the program standard input. Since most mail filters operate this way, the default value is 'Yes'. If you set this option to 'No', read the description of the 'Options' command..

4. Options:
Parameter: String (program arguments)
Default value: None
This options set the arguments to be passed to the postfilter program. If you set InputStdin to 'No' you probably need to specify the path to the mail message to the filter in his options string. This is done substituting the flag '%M' in this string by the actual path of the mail message. For example, if you are using a filter that don't accept mails in his standart input but need that mail to be specified after a '-f' switch, this command argument would be: Options: -f %M.

5. KillerReturnValues:
Parameter: Comma separated list of numbers
Default value: None
This options specifies a comma separated list of numbers (can be a single token) being those numbers return values that in case of matching (one or more of them) against the filter return value will cause the mail to be filtered. You can specify this option and the 'KillerProgramOutput' option at the same time.

6. KillerProgramOutput:
Parameter: String (regular expression)
Default value: None
This option specifies a regular expression that would be checked against the filter program output and in case of match would cause the mail to be filtered. You can specify this option and the 'KillerReturnValues' option at the same time.

7. ShowFilterOutput:
Parameter: Boolean (Yes/No)
Default value: No
This options specifies if the program filter output (stantard output and standard error) must be shown.

8. SavePostFilteredMails:
Parameter: Boolean or String (Yes/No/Output/Stdout)
Default value: No
This options specifies if the mails filtered by the post-filters must be copied to a mbox file ('spambox' file). The possible values are: 'Yes' if the original message must be copied to the spambox, 'No' if don't, 'Output' if the filter program output is what is to be copied to the spambox, instead of the original message and 'Stdout' which does the same that 'Output' but don't include the error output (stderr) which will be printed.

If this option is set to 'Yes' or 'Output' you must also give a value to the option 'PostFilteredMailsMBox' of the <global> options section.

3. Random tips

3.1 Binding Animail filtering to a Mutt macro

Put this in your .muttrc:

macro index X |~/bin/macrospam\n
macro pager X |~/bin/macrospam\n

Now create a new file called macrospam in ~/bin with the text:

#!/bin/bash
grep 'From:' $s | grep -v 'Resent'>> ~/.animailfilters

and make it executable with:

chmod +x ~bin/macrospam

Now, pressing X in Mutt will add the currently selected mail to the Animail filterfile.




Versión 1.1.30

© 1998-2002 Juanjo Álvarez <juanjux@yahoo.es>

Yes, my english is really bad[TM]