ChangeLog

# Revision 1.206  2015/09/21 16:39:51  customdesigned
# Missed check for missing self.dkim_domain
#
# Revision 1.205  2015/05/16 22:36:25  customdesigned
# Pass test cases with access_file_nulls support.
#
# Revision 1.204  2015/05/15 02:08:43  customdesigned
# Log error opening access file.  Support WHITELIST for SMTP-Auth.
#
# Revision 1.203  2015/04/13 03:29:37  customdesigned
# Move more config variables to Config object.  Include added headers
# in check_spam and quarantine.  Update header triage for pydspam-1.3.
#
# Revision 1.202  2014/03/22 19:15:03  customdesigned
# Fix bandomain with no DKIM
#
# Revision 1.201  2014/03/01 05:19:08  customdesigned
# Release 0.8.18-3
#
# Revision 1.200  2013/07/17 21:27:02  customdesigned
# Add logic to mask IPs or use authenticated domain for greylisting.
# Should be a config option, however.
#
# Revision 1.199  2013/05/15 17:23:41  customdesigned
# Support DKIM whitelisting
#
# Revision 1.198  2013/04/30 23:06:47  customdesigned
# Add helofail web template failure message and add common template for L&F.
#
# Revision 1.197  2013/04/25 18:23:09  customdesigned
# auto_whitelist, blacklist were loading from wrong directory.
#
# Revision 1.196  2013/03/27 02:21:30  customdesigned
# Recognize IPv6 localhost.
#
# Revision 1.195  2013/03/17 17:43:42  customdesigned
# Default logdir to datadir.
#
# Revision 1.194  2013/03/15 23:04:38  customdesigned
# Move many configs to datadir
#
# Revision 1.193  2013/03/12 03:29:55  customdesigned
# Add SMTP AUTH test case for bms milter.
#
# Revision 1.192  2013/03/12 01:33:49  customdesigned
# Tab nanny
#
# Revision 1.191  2013/03/09 23:51:11  customdesigned
# Move email_providers to config.  Move many other configs to Config object.
#
# Revision 1.185  2012/07/13 21:05:57  customdesigned
# Don't check banned ips on submission port (587).
#
# Revision 1.184  2012/05/24 18:26:43  customdesigned
# Log unknown commands.
#
# Revision 1.181  2012/04/19 23:20:05  customdesigned
# Simple DKIM signing support
#
# Revision 1.180  2012/04/12 05:37:25  customdesigned
# Skip greylisting for trusted forwarders
#
# Revision 1.179  2012/02/25 22:21:16  customdesigned
# Support urls with fuller explanation for rejections.
#
# Revision 1.178  2011/11/05 16:05:08  customdesigned
# Change openspf.org -> openspf.net
#
# Revision 1.177  2011/11/01 17:43:33  customdesigned
# Trust trusted relay not to be a zombie.
#
# Revision 1.176  2011/10/03 20:07:16  customdesigned
# Make wiretap use orig_from (set efrom to orig_from earlier).
#
# Revision 1.175  2011/10/03 20:01:00  customdesigned
# Let NOTIFY suppress real DSN.  Since notify is forged on forged email,
# perhaps this should be an option.
#
# Revision 1.174  2011/10/03 17:44:38  customdesigned
# Fix SPF fail ip reported for IP6
#
# Revision 1.173  2011/06/16 20:54:48  customdesigned
# Update to pydkim-0.4 and log verified domains
#
# Revision 1.172  2011/06/07 22:24:38  customdesigned
# Remove leftover tempname in envfrom.  Save failed DKIM
#
# Revision 1.171  2011/06/07 19:45:01  customdesigned
# Check DKIM (log only)
#
# Revision 1.170  2011/05/18 02:50:54  customdesigned
# Improve chinese detection
#
# Revision 1.169  2011/04/13 19:50:04  customdesigned
# Move persistent data to /var/lib/milter
#
# Revision 1.168  2011/04/01 02:34:38  customdesigned
# Fix efrom and umis with delayed reject.
#
# Revision 1.166  2011/03/05 05:12:55  customdesigned
# Release 0.8.15
#
# Revision 1.165  2011/03/03 21:45:24  customdesigned
# Extract original MFROM from SRS
#
# Revision 1.164  2010/10/27 03:07:33  customdesigned
# Whitelist recipients from signed MFROM if aborted before DATA.
#
# Revision 1.163  2010/10/16 21:23:00  customdesigned
# Send auto-whitelist recipients to whitelist_mx
#
# Revision 1.162  2010/08/18 03:58:06  customdesigned
# Fix typos
#
# Revision 1.161  2010/08/18 03:52:09  customdesigned
# Update reputation of parsed Received-SPF header if no Gossip header.
#
# Revision 1.160  2010/06/04 03:50:28  customdesigned
# Allow wildcards just above TLD.
#
# Revision 1.159  2010/05/27 21:22:41  customdesigned
# Fix helo policy lookup
#
# Revision 1.158  2010/05/27 18:23:33  customdesigned
# Support HELO policies.
#
# Revision 1.157  2010/05/22 03:57:17  customdesigned
# Bandomain aliases to ban wildcards.
#
# Revision 1.155  2010/04/16 19:51:05  customdesigned
# Max_demerits config to disable banning ips.
#
# Revision 1.154  2010/04/09 18:37:53  customdesigned
# Best guess pass is good enough to get quarrantine DSN or get banned.
#
# Revision 1.153  2010/04/09 18:23:34  customdesigned
# Don't ban just for repeated anonymous MFROM
#
# Revision 1.152  2010/02/15 21:02:29  customdesigned
# Lower reputation bar to avoid greylisting.
#
# Revision 1.151  2009/12/31 19:23:18  customdesigned
# Don't check From unless dspam enabled.
#
# Revision 1.150  2009/12/30 20:53:20  customdesigned
# Require pymilter >= 0.9.3
#
# Revision 1.149  2009/09/14 14:59:53  customdesigned
# Allow illegal HELO from internal to accomodate broken copier firmware.
#
# Revision 1.148  2009/09/14 14:28:22  customdesigned
# Heuristically recognize multiple MXs.
#
# Revision 1.147  2009/09/14 14:24:11  customdesigned
# Trust 127.0.0.1 not to be a zombie
#
# Revision 1.146  2009/08/29 03:38:27  customdesigned
# Don't ban domains unless gossip score available.
#
# Revision 1.145  2009/08/27 21:18:16  customdesigned
# Track banned domains and expand the offenses that can ban an IP.
#
# Revision 1.141  2009/05/20 02:48:18  customdesigned
# Restrict internal DSNs to official MTAs.
#
# Revision 1.140  2009/02/04 02:40:14  customdesigned
# Parse gossip header before add_spam.  Replace nulls in smtp error txt.
# Default internal_policy flag false.
#
# Revision 1.139  2008/12/22 02:34:39  customdesigned
# Fix internal policy
#
# Revision 1.138  2008/12/13 21:22:51  customdesigned
# Split off pymilter
#
# Revision 1.137  2008/12/06 21:13:57  customdesigned
# Fix some reject messages.
#
# Revision 1.136  2008/12/04 19:42:46  customdesigned
# SPF Pass policy
#
# Revision 1.135  2008/10/23 19:58:06  customdesigned
# Example config had different names than actual code :-)
#
# Revision 1.134  2008/10/11 15:45:46  customdesigned
# Don't greylist DSNs.
#
# Revision 1.133  2008/10/09 18:44:54  customdesigned
# Skip greylisting for good reputation.
#
# Revision 1.132  2008/10/09 00:55:13  customdesigned
# Don't reset greylist timer on early retries.
#
# Revision 1.131  2008/10/08 04:57:28  customdesigned
# Greylisting
#
# Revision 1.130  2008/10/02 03:19:00  customdesigned
# Delay strike3 REJECT and don't reject if whitelisted.
# Recognize vacation messages as autoreplies.
#
# Revision 1.129  2008/09/09 23:24:56  customdesigned
# Never ban a trusted relay.
#
# Revision 1.128  2008/09/09 23:08:16  customdesigned
# Wasn't reading banned_ips
#
# Revision 1.127  2008/08/25 18:32:22  customdesigned
# Handle missing gossip_node so self tests pass.
#
# Revision 1.126  2008/08/18 17:47:57  customdesigned
# Log rcpt for SRS rejections.
#
# Revision 1.125  2008/08/06 00:52:38  customdesigned
# CBV policy sends no DSN.  DSN policy sends DSN.
#
# Revision 1.124  2008/08/05 18:04:06  customdesigned
# Send quarantine DSN to SPF PASS only.
#
# Revision 1.123  2008/07/29 21:59:29  customdesigned
# Parse ESMTP params
#
# Revision 1.122  2008/05/08 21:35:56  customdesigned
# Allow explicitly whitelisted email from banned_users.
#
# Revision 1.121  2008/04/10 14:59:35  customdesigned
# Configure gossip TTL.
#
# Revision 1.120  2008/04/02 18:59:14  customdesigned
# Release 0.8.10
#
# Revision 1.119  2008/04/01 00:13:10  customdesigned
# Do not CBV whitelisted addresses.  We already know they are good.
#
# Revision 1.118  2008/01/09 20:15:49  customdesigned
# Handle unquoted fullname when parsing email.
#
# Revision 1.117  2007/11/29 14:35:17  customdesigned
# Packaging tweaks.
#
# Revision 1.116  2007/11/01 20:09:14  customdesigned
# Support temperror policy in access.
#
# Revision 1.115  2007/10/10 18:23:54  customdesigned
# Send quarantine DSN to SPF pass (official or guessed) only.
# Reject blacklisted email too big for dspam.
#
# Revision 1.114  2007/10/10 18:07:50  customdesigned
# Check porn keywords in From header field.
#
# Revision 1.113  2007/09/25 16:37:26  customdesigned
# Tested on RH7
#
# Revision 1.112  2007/09/13 14:51:03  customdesigned
# Report domain on reputation reject.
#
# Revision 1.111  2007/07/25 17:14:59  customdesigned
# Move milter apps to /usr/lib/pymilter
#
# Revision 1.110  2007/07/02 03:06:10  customdesigned
# Ban ips on bad mailfrom offenses as well as bad rcpts.
#
# Revision 1.109  2007/06/23 20:53:05  customdesigned
# Ban IPs based on too many invalid recipients in a connection.  Requires
# configuring check_user.  Tighten HELO best_guess policy.
#
# Revision 1.108  2007/04/19 16:02:43  customdesigned
# Do not process valid SRS recipients as delayed_failure.
#
# Revision 1.107  2007/04/15 01:01:13  customdesigned
# Ban ips with too many bad rcpts on a connection.
#
# Revision 1.105  2007/04/13 17:20:09  customdesigned
# Check access_file at startup.  Compress rcpt to log.
#
# Revision 1.104  2007/04/05 17:59:07  customdesigned
# Stop querying gossip server twice.
#
# Revision 1.103  2007/04/02 18:37:25  customdesigned
# Don't disable gossip for temporary error.
#
# Revision 1.102  2007/03/30 18:13:41  customdesigned
# Report bestguess and helo-spf as key-value pairs in Received-SPF
# instead of in their own headers.
#
# Revision 1.101  2007/03/29 03:06:10  customdesigned
# Don't count DSN and unqualified MAIL FROM as internal_domain.
#
# Revision 1.100  2007/03/24 00:30:24  customdesigned
# Do not CBV for internal domains.
# Revision 1.99  2007/03/23 22:39:10  customdesigned
# Get SMTP-Auth policy from access_file.
#
# Revision 1.98  2007/03/21 04:02:13  customdesigned
# Properly log From: and Sender:
#
# Revision 1.97  2007/03/18 02:32:21  customdesigned
# Gossip configuration options: client or standalone with optional peers.
#
# Revision 1.96  2007/03/17 21:22:48  customdesigned
# New delayed DSN pattern.  Retab (expandtab).
#
# Revision 1.95  2007/03/03 19:18:57  customdesigned
# Fix continuing findsrs when srs.reverse fails.
#
# Revision 1.94  2007/03/03 18:46:26  customdesigned
# Improve delayed failure detection.
#
# Revision 1.93  2007/02/07 23:21:26  customdesigned
# Use re for auto-reply recognition.
#
# Revision 1.92  2007/01/26 03:47:23  customdesigned
# Handle null in header value.
#
# Revision 1.91  2007/01/25 22:47:25  customdesigned
# Persist blacklisting from delayed DSNs.
#
# Revision 1.90  2007/01/23 19:46:20  customdesigned
# Add private relay.
#
# Revision 1.89  2007/01/22 02:46:01  customdesigned
# Convert tabs to spaces.
#
# Revision 1.88  2007/01/19 23:31:38  customdesigned
# Move parse_header to Milter.utils.
# Test case for delayed DSN parsing.
# Fix plock when source missing or cannot set owner/group.
#
# Revision 1.87  2007/01/18 16:48:44  customdesigned
# Doc update.
# Parse From header for delayed failure detection.
# Don't check reputation of trusted host.
# Track IP reputation only when missing PTR.
#
# Revision 1.86  2007/01/16 05:17:29  customdesigned
# REJECT after data for blacklisted emails - so in case of mistakes, a
# legitimate sender will know what happened.
#
# Revision 1.85  2007/01/11 04:31:26  customdesigned
# Negative feedback for bad headers.  Purge cache logs on startup.
#
# Revision 1.84  2007/01/10 04:44:25  customdesigned
# Documentation updates.
#
# Revision 1.83  2007/01/08 23:20:54  customdesigned
# Get user feedback.
#
# Revision 1.82  2007/01/06 04:21:30  customdesigned
# Add config file to spfmilter
#
# Revision 1.81  2007/01/05 23:33:55  customdesigned
# Make blacklist an AddrCache
#
# Revision 1.80  2007/01/05 23:12:12  customdesigned
# Move parse_addr, iniplist, ip4re to Milter.utils
#
# Revision 1.79  2007/01/05 21:25:40  customdesigned
# Move AddrCache to Milter package.
#
# Revision 1.78  2007/01/04 18:01:10  customdesigned
# Do plain CBV when template missing.
#
# Revision 1.77  2006/12/31 03:07:20  customdesigned
# Use HELO identity if good when MAILFROM is bad.
#
# Revision 1.76  2006/12/30 18:58:53  customdesigned
# Skip reputation/whitelist/blacklist when rejecting on SPF.  Add X-Hello-SPF.
#
# Revision 1.75  2006/12/28 01:54:32  customdesigned
# Reject on bad_reputation or blacklist and nodspam.  Match valid helo like
# PTR for guessed SPF pass.
#
# Revision 1.74  2006/12/19 00:59:30  customdesigned
# Add archive option to wiretap.
#
# Revision 1.73  2006/12/04 18:47:03  customdesigned
# Reject multiple recipients to DSN.
# Auto-disable gossip on DB error.
#
# Revision 1.72  2006/11/22 16:31:22  customdesigned
# SRS domains were missing srs_reject check when SES was active.
#
# Revision 1.71  2006/11/22 01:03:28  customdesigned
# Replace last use of deprecated rfc822 module.
#
# Revision 1.70  2006/11/21 18:45:49  customdesigned
# Update a use of deprecated rfc822.  Recognize report-type=delivery-status
# Revision 1.69  2006/11/04 22:09:39  customdesigned
# Another lame DSN heuristic.  Block PTR cache poisoning attack.
#
# Revision 1.68  2006/10/04 03:46:01  customdesigned
# Fix defaults.
#
# Revision 1.67  2006/10/01 01:44:06  customdesigned
# case_sensitive_localpart option, more delayed bounce heuristics,
# optional smart_alias section.
#
# Revision 1.66  2006/07/26 16:42:26  customdesigned
# Support CBV timeout
#
# Revision 1.65  2006/06/21 22:22:00  customdesigned
# Handle multi-line headers in delayed dsns.
#
# Revision 1.64  2006/06/21 21:12:04  customdesigned
# More delayed reject token headers.
# Don't require HELO pass for CBV.
#
# Revision 1.63  2006/05/21 03:41:44  customdesigned
# Fail dsn
#
# Revision 1.61  2006/05/17 21:28:07  customdesigned
# Create GOSSiP record only when connection will procede to DATA.
#
# Revision 1.60  2006/05/12 16:14:48  customdesigned
# Don't require SPF pass for white/black listing mail from trusted relay.
# Support localpart wildcard for white and black lists.
#
# Revision 1.59  2006/04/06 18:14:17  customdesigned
# Check whitelist/blacklist even when not checking SPF (e.g. trusted relay).
#
# Revision 1.58  2006/03/10 20:52:49  customdesigned
# Use re to recognize failure DSNs.
#
# Revision 1.57  2006/03/07 20:50:54  customdesigned
# Use signed Message-ID in delayed reject to blacklist senders
#
# Revision 1.56  2006/02/24 02:12:54  customdesigned
# Properly report hard PermError (lax mode fails also) by always setting
# perm_error attribute with PermError exception.  Improve reporting of
# invalid domain PermError.
#
# Revision 1.55  2006/02/17 05:04:29  customdesigned
# Use SRS sign domain list.
# Accept but do not use for training whitelisted senders without SPF pass.
# Immediate rejection of unsigned bounces.
#
# Revision 1.54  2006/02/16 02:16:36  customdesigned
# User specific SPF receiver policy.
#
# Revision 1.53  2006/02/12 04:15:01  customdesigned
# Remove spf dependency for iniplist
#
# Revision 1.52  2006/02/12 02:12:08  customdesigned
# Use CIDR notation for internal connect list.
#
# Revision 1.51  2006/02/12 01:13:58  customdesigned
# Don't check rcpt user list when signed MFROM.
#
# Revision 1.50  2006/02/09 20:39:43  customdesigned
# Use CIDR notation for trusted_relay iplist
#
# Revision 1.49  2006/01/30 23:14:48  customdesigned
# put back eom condition
#
# Revision 1.48  2006/01/12 20:31:24  customdesigned
# Accelerate training via whitelist and blacklist.
#
# Revision 1.47  2005/12/29 04:49:10  customdesigned
# Do not auto-whitelist autoreplys
#
# Revision 1.46  2005/12/28 20:17:29  customdesigned
# Expire and renew AddrCache entries
#
# Revision 1.45  2005/12/23 22:34:46  customdesigned
# Put guessed result in separate header.
#
# Revision 1.44  2005/12/23 21:47:07  customdesigned
# Move Received-SPF header to top.
#
# Revision 1.43  2005/12/09 16:54:01  customdesigned
# Select neutral DSN template for best_guess
#
# Revision 1.42  2005/12/01 22:42:32  customdesigned
# improve gossip support.
# Initialize srs_domain from srs.srs config property.  Should probably
# always block unsigned DSN when signing all.
#
# Revision 1.41  2005/12/01 18:59:25  customdesigned
# Fix neutral policy.  pobox.com -> openspf.org
#
# Revision 1.40  2005/11/07 21:22:35  customdesigned
# GOSSiP support, local database only.
#
# Revision 1.39  2005/10/31 00:04:58  customdesigned
# Simple implementation of trusted_forwarder list.  Inefficient for
# more than 1 or 2 entries.
#
# Revision 1.38  2005/10/28 19:36:54  customdesigned
# Don't check internal_domains for trusted_relay.
#
# Revision 1.37  2005/10/28 09:30:49  customdesigned
# Do not send quarantine DSN when sender is DSN.
#
# Revision 1.36  2005/10/23 16:01:29  customdesigned
# Consider MAIL FROM a match for supply_sender when a subdomain of From or Sender
#
# Revision 1.35  2005/10/20 18:47:27  customdesigned
# Configure auto_whitelist senders.
#
# Revision 1.34  2005/10/19 21:07:49  customdesigned
# access.db stores keys in lower case
#
# Revision 1.33  2005/10/19 19:37:50  customdesigned
# Train screener on whitelisted messages.
#
# Revision 1.32  2005/10/14 16:17:31  customdesigned
# Auto whitelist refinements.
#
# Revision 1.31  2005/10/14 01:14:08  customdesigned
# Auto whitelist feature.
#
# Revision 1.30  2005/10/12 16:36:30  customdesigned
# Release 0.8.3
#
# Revision 1.29  2005/10/11 22:50:07  customdesigned
# Always check HELO except for SPF pass, temperror.
#
# Revision 1.28  2005/10/10 23:50:20  customdesigned
# Use logging module to make logging threadsafe (avoid splitting log lines)
#
# Revision 1.27  2005/10/10 20:15:33  customdesigned
# Configure SPF policy via sendmail access file.
#
# Revision 1.26  2005/10/07 03:23:40  customdesigned
# Banned users option.  Experimental feature to supply Sender when
# missing and MFROM domain doesn't match From.  Log cipher bits for
# SMTP AUTH.  Sketch access file feature.
#
# Revision 1.25  2005/09/08 03:55:08  customdesigned
# Handle perverse MFROM quoting.
#
# Revision 1.24  2005/08/18 03:36:54  customdesigned
# Don't innoculate with SCREENED mail.
#
# Revision 1.23  2005/08/17 19:35:27  customdesigned
# Send DSN before adding message to quarantine.
#
# Revision 1.22  2005/08/11 22:17:58  customdesigned
# Consider SMTP AUTH connections internal.
#
# Revision 1.21  2005/08/04 21:21:31  customdesigned
# Treat fail like softfail for selected (braindead) domains.
# Treat mail according to extended processing results, but
# report any PermError that would officially result via DSN.
#
# Revision 1.20  2005/08/02 18:04:35  customdesigned
# Keep screened honeypot mail, but optionally discard honeypot only mail.
#
# Revision 1.19  2005/07/20 03:30:04  customdesigned
# Check pydspam version for honeypot, include latest pyspf changes.
#
# Revision 1.18  2005/07/17 01:25:44  customdesigned
# Log as well as use extended result for best guess.
#
# Revision 1.17  2005/07/15 20:25:36  customdesigned
# Use extended results processing for best_guess.
#
# Revision 1.16  2005/07/14 03:23:33  customdesigned
# Make SES package optional.  Initial honeypot support.
#
# Revision 1.15  2005/07/06 04:05:40  customdesigned
# Initial SES integration.
#
# Revision 1.14  2005/07/02 23:27:31  customdesigned
# Don't match hostnames for internal connects.
#
# Revision 1.13  2005/07/01 16:30:24  customdesigned
# Always log trusted Received and Received-SPF headers.
#
# Revision 1.12  2005/06/20 22:35:35  customdesigned
# Setreply for rejectvirus.
#
# Revision 1.11  2005/06/17 02:07:20  customdesigned
# Release 0.8.1
#
# Revision 1.10  2005/06/16 18:35:51  customdesigned
# Ignore HeaderParseError decoding header
#
# Revision 1.9  2005/06/14 21:55:29  customdesigned
# Check internal_domains for outgoing mail.
#
# Revision 1.8  2005/06/06 18:24:59  customdesigned
# Properly log exceptions from pydspam
#
# Revision 1.7  2005/06/04 19:41:16  customdesigned
# Fix bugs from testing RPM
#
# Revision 1.6  2005/06/03 04:57:05  customdesigned
# Organize config reader by section.  Create defang section.
#
# Revision 1.5  2005/06/02 15:00:17  customdesigned
# Configure banned extensions.  Scan zipfile option with test case.
#
# Revision 1.4  2005/06/02 04:18:55  customdesigned
# Update copyright notices after reading article on /.
#
# Revision 1.3  2005/06/02 02:09:00  customdesigned
# Record timestamp in send_dsn.log
#
# Revision 1.2  2005/06/02 01:00:36  customdesigned
# Support configurable templates for DSNs.