From jesse.thompson at doit.wisc.edu Tue Apr 10 20:05:22 2007 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Tue, 10 Apr 2007 12:05:22 -0500 Subject: [Gross] first post! Message-ID: <461BC3D2.4000200@doit.wisc.edu> yes, this is much better than the Google Groups/Mail silo. Gross rocks! Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3304 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Wed Apr 11 20:32:48 2007 From: eino at utu.fi (Eino Tuominen) Date: Wed, 11 Apr 2007 20:32:48 +0300 Subject: [Gross] First user complaint! Message-ID: <461D1BC0.90003@utu.fi> Hi, Finally, the first user complaint about delayed mail. Mail relays of iki.fi domain ended up on dnsbl.sorbs.net. Quote from their pages: The Internet Users Forever IKI is a non-profit society which provides its members, private individuals in Finland, permanent iki.fi-addresses with e-mail and WWW forwarding services Btw, I've been a member for ages, hence my iki-address et at iki.fi. As you guessed, we have a lot of users who forward their iki-mail to us. But, the problem eventually was on their end - it took a good 15 hours for their Sendmail system to retry each delivery... But, it took a whole year until Gross was (a minor) part of a problem. -- Eino Tuominen From jesse.thompson at doit.wisc.edu Wed Apr 11 21:12:15 2007 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Wed, 11 Apr 2007 13:12:15 -0500 Subject: [Gross] First user complaint! In-Reply-To: <461D1BC0.90003@utu.fi> References: <461D1BC0.90003@utu.fi> Message-ID: <461D24FF.6000501@doit.wisc.edu> That's cool. Gross blocks millions of messages per day here and I've yet to hear a single complaint. I don't think anyone even notices, other than the fact that they marvel how they have 50% less mail in their Junk Mail folders. Eino Tuominen wrote: > Hi, > > Finally, the first user complaint about delayed mail. Mail relays of > iki.fi domain ended up on dnsbl.sorbs.net. > > Quote from their pages: > The Internet Users Forever IKI is a non-profit society which > provides its members, private individuals in Finland, permanent > iki.fi-addresses with e-mail and WWW forwarding services > > Btw, I've been a member for ages, hence my iki-address et at iki.fi. As you > guessed, we have a lot of users who forward their iki-mail to us. But, > the problem eventually was on their end - it took a good 15 hours for > their Sendmail system to retry each delivery... > > But, it took a whole year until Gross was (a minor) part of a problem. > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3304 bytes Desc: S/MIME Cryptographic Signature URL: From jesse.thompson at doit.wisc.edu Mon Apr 16 17:50:41 2007 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Mon, 16 Apr 2007 09:50:41 -0500 Subject: [Gross] greylisting on reverse dns patterns Message-ID: <46238D41.2050407@doit.wisc.edu> Hi, We're seeing an uptick in spam here. Gross is still blocking over 58% (6% match) of the messages, but the increase in overall spam is becoming more noticeable. I'm noticing that a lot of the spam is originating from IP addresses that have a reverse dns record that indicates that the IP is dynamically assigned. e.g. "pool" or "dynamic" or "dhcp" Is there a more aggressive RBL that will list IPs that are on known dynamic networks? Here is the list of RBLs that I'm currently using. dnsbl = rbl-plus.mail-abuse.org dnsbl = bl.spamcop.net dnsbl = dnsbl.njabl.org dnsbl = cbl.abuseat.org dnsbl = dnsbl.sorbs.net dnsbl = list.dsbl.org dnsbl = multihop.dsbl.org dnsbl = zen.spamhaus.org What about adding a feature to Gross to match on the reverse dns of the client_ip? I'm considering cracking open the source code and dusting off my C reference to consider implementing this feature myself. Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3304 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Mon Apr 16 19:42:24 2007 From: eino at utu.fi (Eino Tuominen) Date: Mon, 16 Apr 2007 19:42:24 +0300 Subject: [Gross] greylisting on reverse dns patterns In-Reply-To: <46238D41.2050407@doit.wisc.edu> References: <46238D41.2050407@doit.wisc.edu> Message-ID: <4623A770.3080400@utu.fi> Jesse Thompson wrote: > Hi, > > We're seeing an uptick in spam here. Gross is still blocking over 58% > (6% match) of the messages, but the increase in overall spam is becoming > more noticeable. I'm noticing that a lot of the spam is originating > from IP addresses that have a reverse dns record that indicates that the > IP is dynamically assigned. e.g. "pool" or "dynamic" or "dhcp" > > Is there a more aggressive RBL that will list IPs that are on known > dynamic networks? Here is the list of RBLs that I'm currently using. > > dnsbl = rbl-plus.mail-abuse.org > dnsbl = bl.spamcop.net > dnsbl = dnsbl.njabl.org > dnsbl = cbl.abuseat.org > dnsbl = dnsbl.sorbs.net > dnsbl = list.dsbl.org > dnsbl = multihop.dsbl.org > dnsbl = zen.spamhaus.org We have dnsbl=rbl-plus.mail-abuse.org dnsbl=bl.spamcop.net dnsbl=combined.njabl.org dnsbl=cbl.abuseat.org dnsbl=dnsbl.sorbs.net check = dnsbl check = blocker Remember that you have to pay for spamhaus lists. I incorrectly had spamhaus lists in the early example configs as I didn't know it. Current stats: 22/1/77 (trust/match/grey). I have always wondered why Gross is more effective with our mail flow - it would be interesting to know why. > What about adding a feature to Gross to match on the reverse dns of the > client_ip? I'm considering cracking open the source code and dusting > off my C reference to consider implementing this feature myself. There are numerous good regexp patterns to find dynamic pools - I think you can find some with Google. Unfortunately I have no good pointers in my bookmarks. It should be rather straight forward to write that kind of a check. And, as I have mentioned before, other good candidates for blocking are messages with sender domain MX pointing to 127.0.0.1. That needs the modifications I have been (not so busy) writing in order to get that envelope sender address trough to the checks. And, I think I will implement the check delays I have told you before. The blocker check is much cheaper than a bunch of dns queries - Gross should perform the expensive checks only if cheaper checks fail. -- Eino Tuominen From eino at utu.fi Mon Apr 16 19:48:42 2007 From: eino at utu.fi (Eino Tuominen) Date: Mon, 16 Apr 2007 19:48:42 +0300 Subject: [Gross] greylisting on reverse dns patterns In-Reply-To: <4623A770.3080400@utu.fi> References: <46238D41.2050407@doit.wisc.edu> <4623A770.3080400@utu.fi> Message-ID: <4623A8EA.4060001@utu.fi> Eino Tuominen wrote: > > There are numerous good regexp patterns to find dynamic pools - I think > you can find some with Google. Unfortunately I have no good pointers in > my bookmarks. It should be rather straight forward to write that kind of Check this out: http://archives.neohapsis.com/archives/postfix/2006-11/0402.html -- Eino Tuominen From R.E.Sonneveld at sonnection.nl Thu Apr 19 00:45:15 2007 From: R.E.Sonneveld at sonnection.nl (Rolf E. Sonneveld) Date: Wed, 18 Apr 2007 23:45:15 +0200 Subject: [Gross] New member Message-ID: <4626916B.80300@sonnection.nl> Hi, my name is Rolf Sonneveld, I'm new to this list. I have implemented gross for one of my customers, and will deploy it possibly in the future for an ISP as well. I very much hope that gross gets more and more interest from people and that it will more and more installed base. /rolf From eino at utu.fi Fri Apr 20 23:08:04 2007 From: eino at utu.fi (Eino Tuominen) Date: Fri, 20 Apr 2007 23:08:04 +0300 Subject: [Gross] New features in svn trunk Message-ID: <46291DA4.6000808@utu.fi> Hi, I just committed some modifications to svn trunk. They include - code cleanup and a bug fix in thread_pool.c - milter implementation - support for definitive checks Jesse pointed out an issue: due to random (seems to be, I have not checked) scheduling of threads waiting on pthread_cond_wait() on Linux threads never shut down as they almost never had to wait more than a few seconds for new work. It's now fixed and I did some code cleanup doing that, too. Milter is configured with --enable-milter, config example is included in config file. The only definitive check so far is 'random' - the check is just for testing purposes only. These modifications are not tested in production, so be extra careful should you put it on test. Happy testing! -- Eino Tuominen From R.E.Sonneveld at sonnection.nl Fri Apr 20 23:29:23 2007 From: R.E.Sonneveld at sonnection.nl (Rolf E. Sonneveld) Date: Fri, 20 Apr 2007 22:29:23 +0200 Subject: [Gross] New features in svn trunk In-Reply-To: <46291DA4.6000808@utu.fi> References: <46291DA4.6000808@utu.fi> Message-ID: <462922A3.5090400@sonnection.nl> Eino Tuominen wrote: > Hi, > > I just committed some modifications to svn trunk. They include > > - code cleanup and a bug fix in thread_pool.c > - milter implementation > - support for definitive checks > > Jesse pointed out an issue: due to random (seems to be, I have not > checked) scheduling of threads waiting on pthread_cond_wait() on Linux > threads never shut down as they almost never had to wait more than a few > seconds for new work. It's now fixed and I did some code cleanup doing > that, too. > > Milter is configured with --enable-milter, config example is included in > config file. > > The only definitive check so far is 'random' - the check is just for > testing purposes only. > > These modifications are not tested in production, so be extra careful > should you put it on test. > > Happy testing! > Thanks for the update and fixes. regards, /rolf From eino at utu.fi Tue May 15 15:37:20 2007 From: eino at utu.fi (Eino Tuominen) Date: Tue, 15 May 2007 15:37:20 +0300 Subject: [Gross] New snapshot release Message-ID: <4649A980.10108@utu.fi> Hi all, I uploaded a new snapshot release, gross-svn-r206.tar.gz. It includes right hand side and whitelist checks, milter implementation and a few other improvements under the hood. Please, give it a go and report all oddities. -- Eino Tuominen From eino at utu.fi Thu May 24 12:42:05 2007 From: eino at utu.fi (Eino Tuominen) Date: Thu, 24 May 2007 12:42:05 +0300 Subject: [Gross] New snapshot available Message-ID: <46555DED.8050205@utu.fi> Hello all, Last development snapshot contained a couple of memory leaks. The new version fixes those - no other changes. I have used r206 in production since its release. Memory leaks aside, I have not had any problems with it. RHSBL does not seem to be very efficient, though. Have you had any luck with the (still rather new) milter implementation? -- Eino Tuominen From R.E.Sonneveld at sonnection.nl Thu May 24 13:13:10 2007 From: R.E.Sonneveld at sonnection.nl (Rolf E. Sonneveld) Date: Thu, 24 May 2007 12:13:10 +0200 Subject: [Gross] New snapshot available In-Reply-To: <46555DED.8050205@utu.fi> References: <46555DED.8050205@utu.fi> Message-ID: <46556536.3080107@sonnection.nl> Eino Tuominen wrote: > Hello all, > > Last development snapshot contained a couple of memory leaks. The new > version fixes those - no other changes. > > I have used r206 in production since its release. Memory leaks aside, I > have not had any problems with it. RHSBL does not seem to be very > efficient, though. > > Have you had any luck with the (still rather new) milter implementation? > I haven't had the chance to implement the new version yet, sorry for that. I'm planning to test the milter implementation the next couple of weeks. Thanks for keeping us up to date and for the development and bugfixing of gross. /rolf From jesse.thompson at doit.wisc.edu Thu May 24 22:06:28 2007 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Thu, 24 May 2007 14:06:28 -0500 Subject: [Gross] New snapshot available In-Reply-To: <46555DED.8050205@utu.fi> References: <46555DED.8050205@utu.fi> Message-ID: <4655E234.9090404@doit.wisc.edu> Eino Tuominen wrote: > Hello all, > > Last development snapshot contained a couple of memory leaks. The new > version fixes those - no other changes. > > I have used r206 in production since its release. Memory leaks aside, I > have not had any problems with it. RHSBL does not seem to be very > efficient, though. I haven't had a chance to install this yet. I'm still off for another week or so. After that I plan to migrate gross to a new server environment. I'll try this release after things settle back down. Can you elaborate on the efficiency problem with the RHSBLs? Is it a latency problem with the remote service, or something local? I wonder if it would be possible for Sophos users to query the URI_CLASS_* spam rules. This might provide better performance similar to the Blocker queries. Although, I don't think that there is an API provided to query these rules on demand. Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3304 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Thu May 24 22:40:12 2007 From: eino at utu.fi (Eino Tuominen) Date: Thu, 24 May 2007 22:40:12 +0300 Subject: [Gross] New snapshot available In-Reply-To: <4655E234.9090404@doit.wisc.edu> References: <46555DED.8050205@utu.fi> <4655E234.9090404@doit.wisc.edu> Message-ID: <4655EA1C.8010606@utu.fi> Jesse Thompson wrote: > > Can you elaborate on the efficiency problem with the RHSBLs? Is it a > latency problem with the remote service, or something local? I didn't put my words very well. I meant to say that there is surprisingly few hits from rhsbl I'm using: 0: Grossd OK. Update queue: 19 (In: 19 + Out: 0) Trust: 381864 Match: 24346 Greylist: 972697 Queries/sec: 1.684859 Dnsbl matches: grossd dnsbl matches (rhsbl.sorbs.net, rbl-plus.mail-abuse.org, bl.spamcop.net, combined.njabl.org, cbl.abuseat.org, dnsbl.sorbs.net): 73, 220039, 266894, 211974, 131848, 140246 Maybe there are better lists than rhsbl.sorbs.net. > I wonder if it would be possible for Sophos users to query the > URI_CLASS_* spam rules. This might provide better performance similar > to the Blocker queries. Although, I don't think that there is an API > provided to query these rules on demand. Interesting idea. It may be possible to do some reverse engineering. I'll see into it. BTW, would you see use of Gross as a spam trap? There could be spam trap addresses hidden on web pages and stuff, and if grossd saw an attempt to deliver mail to a spam trap address, it would add the sending host to a local block list. I'm not sure if it would help much, though... -- Eino Tuominen From eino at utu.fi Mon May 28 15:47:35 2007 From: eino at utu.fi (Eino Tuominen) Date: Mon, 28 May 2007 15:47:35 +0300 Subject: [Gross] SPF rant Message-ID: <465ACF67.9080907@utu.fi> Hi all, I've been trying to finish the SPF check module... I was hoping to use libspf2 as it seemed to be a decent project at the fist glance. But, I'm really having a hard time with it and now I noticed that the documentation and example implementations doesn't look even close the same. So, has any of you come across with any decent SPF implementations before? If not, I'll put the module on shelf to wait on better days. -- Eino Tuominen From R.E.Sonneveld at sonnection.nl Tue May 29 11:05:44 2007 From: R.E.Sonneveld at sonnection.nl (Rolf E. Sonneveld) Date: Tue, 29 May 2007 10:05:44 +0200 Subject: [Gross] SPF rant In-Reply-To: <465ACF67.9080907@utu.fi> References: <465ACF67.9080907@utu.fi> Message-ID: <465BDED8.5070303@sonnection.nl> Eino Tuominen wrote: > Hi all, > > I've been trying to finish the SPF check module... I was hoping to use > libspf2 as it seemed to be a decent project at the fist glance. But, I'm > really having a hard time with it and now I noticed that the > documentation and example implementations doesn't look even close the same. > > So, has any of you come across with any decent SPF implementations > before? Not me, sorry for that. I have watched the developments of SPF and Sender-ID closely, and although many people think SPF is the ultimate solution, it introduces many problems for real world situations. Did you consider to implement DKIM? > If not, I'll put the module on shelf to wait on better days. > /rolf From jesse.thompson at doit.wisc.edu Mon Aug 6 19:48:15 2007 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Mon, 06 Aug 2007 11:48:15 -0500 Subject: [Gross] IP Reputation Message-ID: <46B750CF.9020707@doit.wisc.edu> I've been hearing a lot about IP reputation lately. Not that it's new (SenderBase has been around for a while) but it seems to be trendy. Basically, it's a score of the sender based on relative volumes and spam ratios. I wonder if it would be possible to use this type of information in gross. i.e. set certain reputation score thresholds for certain actions. I see that Sendmail has Flow Control, and IronPort has SenderBase. Does anyone know of a standalone IP reputation service that's worth considering integrating into gross? Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Mon Aug 6 22:30:29 2007 From: eino at utu.fi (Eino Tuominen) Date: Mon, 06 Aug 2007 22:30:29 +0300 Subject: [Gross] IP Reputation In-Reply-To: <46B750CF.9020707@doit.wisc.edu> References: <46B750CF.9020707@doit.wisc.edu> Message-ID: <46B776D5.9070706@utu.fi> Jesse Thompson wrote: > I've been hearing a lot about IP reputation lately. Not that it's new > (SenderBase has been around for a while) but it seems to be trendy. > Basically, it's a score of the sender based on relative volumes and spam > ratios. I wonder if it would be possible to use this type of > information in gross. i.e. set certain reputation score thresholds for > certain actions. > > I see that Sendmail has Flow Control, and IronPort has SenderBase. Does > anyone know of a standalone IP reputation service that's worth > considering integrating into gross? It should be pretty straightforward to write a check module to query SenderBase with gross. But, many have found it rather unusable. For instance: http://www.mail-archive.com/policyd-weight-list at ek-muc.de/msg00573.html I recall there was an open reputation service initiative, but I don't know if it got air under its wings. -- Eino Tuominen From jesse.thompson at doit.wisc.edu Mon Aug 6 22:40:05 2007 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Mon, 06 Aug 2007 14:40:05 -0500 Subject: [Gross] IP Reputation In-Reply-To: <46B776D5.9070706@utu.fi> References: <46B750CF.9020707@doit.wisc.edu> <46B776D5.9070706@utu.fi> Message-ID: <46B77915.9090606@doit.wisc.edu> Eino Tuominen wrote: > Jesse Thompson wrote: >> I've been hearing a lot about IP reputation lately. Not that it's new >> (SenderBase has been around for a while) but it seems to be trendy. >> Basically, it's a score of the sender based on relative volumes and spam >> ratios. I wonder if it would be possible to use this type of >> information in gross. i.e. set certain reputation score thresholds for >> certain actions. >> >> I see that Sendmail has Flow Control, and IronPort has SenderBase. Does >> anyone know of a standalone IP reputation service that's worth >> considering integrating into gross? > > It should be pretty straightforward to write a check module to query > SenderBase with gross. But, many have found it rather unusable. For > instance: > http://www.mail-archive.com/policyd-weight-list at ek-muc.de/msg00573.html > > I recall there was an open reputation service initiative, but I don't > know if it got air under its wings. > I assumed that SenderBase would not allow direct querying. I was thinking that it could be used to find the bots before they are blacklisted based on their historical [lack of] of email volumes. Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From jesse.thompson at doit.wisc.edu Thu Aug 9 18:16:17 2007 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Thu, 09 Aug 2007 10:16:17 -0500 Subject: [Gross] project status? Message-ID: <46BB2FC1.5000705@doit.wisc.edu> I haven't been paying much attention to gross lately. I was out for a while, then I focussed on migrating gross from our linux servers to solaris zones on our MTAs, after that I was focussed on other things. So, what's the status of this project? Version 0.8.2 has been extremely stable and effective. Is it ready for a 1.0 release? Or is svn-r207 in need of testing first? What enhancements should we prioritize next? -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Fri Aug 10 10:05:05 2007 From: eino at utu.fi (Eino Tuominen) Date: Fri, 10 Aug 2007 10:05:05 +0300 Subject: [Gross] project status? In-Reply-To: <46BB2FC1.5000705@doit.wisc.edu> References: <46BB2FC1.5000705@doit.wisc.edu> Message-ID: <46BC0E21.9040503@utu.fi> Jesse Thompson wrote: > I haven't been paying much attention to gross lately. I was out for a > while, then I focussed on migrating gross from our linux servers to > solaris zones on our MTAs, after that I was focussed on other things. > > So, what's the status of this project? Version 0.8.2 has been extremely > stable and effective. Is it ready for a 1.0 release? Or is svn-r207 in > need of testing first? I have not had any time to work on gross lately. svn-r207 must be tested but it has a bug that causes a memoryleak. I suspect the bug is in the blocker check, but have not had the time to make sure of it. I have to rewrite the blocker check once again to use asynchronous IO because synchronous IO seems to block some times due to glitches in PMX blocker service. The main modifications since 0.8.2 are (from NEWS file): * preliminary support for MILTER protocol, check config file * support for definitive checks: it's now possible to write checks that cause grossd to block or pass (whitelisting) * dnsbl check is now run time configurable * two new check types: RHSBL and DNSWL - check config file for documentation and examples. DNSWL is first definitive check. - improved thread counting in thread_pool.c - code cleaning Issues fixed: #40: new type of check - RHSBL (RFE) #41: new type of check - DNSWL (RFE) #46: Make dnsbl checking run time configurable (RFE) #47: Grossd will not run any checks if configured with --disable-dnsbl > What enhancements should we prioritize next? 1. modify blocker check to use asynchronous I/O, 2. make sure the code is stable. 3. finish the documentation. Then I'll release gross 1.0. After that we'll be able to make more radical changes. Are there any "must have"-features that need to be implemented before 1.0? -- Eino Tuominen From jesse.thompson at doit.wisc.edu Fri Aug 10 15:57:29 2007 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Fri, 10 Aug 2007 07:57:29 -0500 Subject: [Gross] project status? In-Reply-To: <46BC0E21.9040503@utu.fi> References: <46BB2FC1.5000705@doit.wisc.edu> <46BC0E21.9040503@utu.fi> Message-ID: <46BC60B9.2090802@doit.wisc.edu> Nice work, as always. Do you think that the memory leak is severe enough that I won't be able to upgrade to this version in production? Otherwise, I'd like to try out the RHSBL and the DNSWL enhancements. I can't remember if it was ever suggested to add simple DNS checks, like imposing restrictions on the reverse DNS record of the IP. That feature could be pushed out post 1.0 though. I think that your plan on ironing out the last of the stability bugs and finishing the documentation is best. Would you like me to work on some of the documentation? Jesse Eino Tuominen wrote: > Jesse Thompson wrote: >> I haven't been paying much attention to gross lately. I was out for a >> while, then I focussed on migrating gross from our linux servers to >> solaris zones on our MTAs, after that I was focussed on other things. >> >> So, what's the status of this project? Version 0.8.2 has been extremely >> stable and effective. Is it ready for a 1.0 release? Or is svn-r207 in >> need of testing first? > > I have not had any time to work on gross lately. svn-r207 must be tested > but it has a bug that causes a memoryleak. I suspect the bug is in the > blocker check, but have not had the time to make sure of it. > > I have to rewrite the blocker check once again to use asynchronous IO > because synchronous IO seems to block some times due to glitches in PMX > blocker service. > > The main modifications since 0.8.2 are (from NEWS file): > > * preliminary support for MILTER protocol, check config file > * support for definitive checks: it's now possible to write > checks that cause grossd to block or pass (whitelisting) > * dnsbl check is now run time configurable > * two new check types: RHSBL and DNSWL - check config file > for documentation and examples. DNSWL is first definitive > check. > > - improved thread counting in thread_pool.c > - code cleaning > > Issues fixed: > #40: new type of check - RHSBL (RFE) > #41: new type of check - DNSWL (RFE) > #46: Make dnsbl checking run time configurable (RFE) > #47: Grossd will not run any checks if configured with --disable-dnsbl > >> What enhancements should we prioritize next? > > 1. modify blocker check to use asynchronous I/O, > 2. make sure the code is stable. > 3. finish the documentation. > > Then I'll release gross 1.0. After that we'll be able to make more > radical changes. Are there any "must have"-features that need to be > implemented before 1.0? > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From R.E.Sonneveld at sonnection.nl Sat Aug 11 00:01:34 2007 From: R.E.Sonneveld at sonnection.nl (Rolf E. Sonneveld) Date: Fri, 10 Aug 2007 23:01:34 +0200 Subject: [Gross] project status? In-Reply-To: <46BC60B9.2090802@doit.wisc.edu> References: <46BB2FC1.5000705@doit.wisc.edu> <46BC0E21.9040503@utu.fi> <46BC60B9.2090802@doit.wisc.edu> Message-ID: <46BCD22E.7010907@sonnection.nl> Jesse Thompson wrote: > Nice work, as always. > > Do you think that the memory leak is severe enough that I won't be > able to upgrade to this version in production? Otherwise, I'd like to > try out the RHSBL and the DNSWL enhancements. > > I can't remember if it was ever suggested to add simple DNS checks, > like imposing restrictions on the reverse DNS record of the IP. That > feature could be pushed out post 1.0 though. > > I think that your plan on ironing out the last of the stability bugs > and finishing the documentation is best. Would you like me to work on > some of the documentation? > > Jesse > > Eino Tuominen wrote: >> Jesse Thompson wrote: >>> I haven't been paying much attention to gross lately. I was out for a >>> while, then I focussed on migrating gross from our linux servers to >>> solaris zones on our MTAs, after that I was focussed on other things. >>> >>> So, what's the status of this project? Version 0.8.2 has been >>> extremely >>> stable and effective. Is it ready for a 1.0 release? Or is >>> svn-r207 in >>> need of testing first? >> >> I have not had any time to work on gross lately. svn-r207 must be tested >> but it has a bug that causes a memoryleak. I suspect the bug is in the >> blocker check, but have not had the time to make sure of it. >> >> I have to rewrite the blocker check once again to use asynchronous IO >> because synchronous IO seems to block some times due to glitches in PMX >> blocker service. >> >> The main modifications since 0.8.2 are (from NEWS file): >> >> * preliminary support for MILTER protocol, check config file >> * support for definitive checks: it's now possible to write >> checks that cause grossd to block or pass (whitelisting) >> * dnsbl check is now run time configurable >> * two new check types: RHSBL and DNSWL - check config file >> for documentation and examples. DNSWL is first definitive >> check. >> >> - improved thread counting in thread_pool.c >> - code cleaning >> >> Issues fixed: >> #40: new type of check - RHSBL (RFE) >> #41: new type of check - DNSWL (RFE) >> #46: Make dnsbl checking run time configurable (RFE) >> #47: Grossd will not run any checks if configured with --disable-dnsbl >> >>> What enhancements should we prioritize next? >> >> 1. modify blocker check to use asynchronous I/O, >> 2. make sure the code is stable. >> 3. finish the documentation. >> >> Then I'll release gross 1.0. After that we'll be able to make more >> radical changes. Are there any "must have"-features that need to be >> implemented before 1.0? what does the name 'gross' stand for? I believe the 'gr' in gross is for 'greylisting', isn't it? Looking at the developments up till now you may want to reconsider the name of gross, as it's no longer only doing greylisting. It's becoming more and more of a 'Generic Rock-solid Open Source Spamfilter' ;-) And may I suggest for version 1.1 support for DKIM? Regards, /rolf From jesse.thompson at doit.wisc.edu Tue Sep 4 18:43:54 2007 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Tue, 04 Sep 2007 10:43:54 -0500 Subject: [Gross] grossd - blocker TIME_WAIT connections Message-ID: <46DD7D3A.2060700@doit.wisc.edu> version 0.8.2 I've been trying to figure out why I'm been having trouble scping and sshing from my gross servers. One thing we noticed was a large number of connections to the Sophos blocker in TIME_WAIT state. netstat -n | grep TIME_WAIT | grep -c 4466 2317 Is this normal? -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Wed Sep 5 12:09:37 2007 From: eino at utu.fi (Eino Tuominen) Date: Wed, 05 Sep 2007 12:09:37 +0300 Subject: [Gross] grossd - blocker TIME_WAIT connections In-Reply-To: <46DD7D3A.2060700@doit.wisc.edu> References: <46DD7D3A.2060700@doit.wisc.edu> Message-ID: <46DE7251.1090809@utu.fi> Jesse Thompson wrote: > version 0.8.2 > > I've been trying to figure out why I'm been having trouble scping and > sshing from my gross servers. One thing we noticed was a large number > of connections to the Sophos blocker in TIME_WAIT state. > > netstat -n | grep TIME_WAIT | grep -c 4466 > 2317 I think it is. It means that you are making some 100 blocker queries per second. The problem with the current blocker check implementation is that the connections are not reused. So grossd opens a new TCP connection for each blocker query. -- Eino Tuominen From jesse.thompson at doit.wisc.edu Wed Sep 5 14:52:38 2007 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Wed, 05 Sep 2007 06:52:38 -0500 Subject: [Gross] grossd - blocker TIME_WAIT connections In-Reply-To: <46DE7251.1090809@utu.fi> References: <46DD7D3A.2060700@doit.wisc.edu> <46DE7251.1090809@utu.fi> Message-ID: <46DE9886.4020801@doit.wisc.edu> Eino Tuominen wrote: > Jesse Thompson wrote: >> version 0.8.2 >> >> I've been trying to figure out why I'm been having trouble scping and >> sshing from my gross servers. One thing we noticed was a large number >> of connections to the Sophos blocker in TIME_WAIT state. >> >> netstat -n | grep TIME_WAIT | grep -c 4466 >> 2317 > > I think it is. It means that you are making some 100 blocker queries per > second. The problem with the current blocker check implementation is > that the connections are not reused. So grossd opens a new TCP > connection for each blocker query. Ah. ok. The ssh issue isn't a big enough problem to warrant major code changes in grossd. We've got the TIME_WAIT cleanup set to 60 seconds, so we may try turning it down to 30 or 45 to see if it helps. Thanks! Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Wed Sep 5 16:01:32 2007 From: eino at utu.fi (Eino Tuominen) Date: Wed, 05 Sep 2007 16:01:32 +0300 Subject: [Gross] grossd - blocker TIME_WAIT connections In-Reply-To: <46DE9886.4020801@doit.wisc.edu> References: <46DD7D3A.2060700@doit.wisc.edu> <46DE7251.1090809@utu.fi> <46DE9886.4020801@doit.wisc.edu> Message-ID: <46DEA8AC.1020709@utu.fi> Jesse Thompson wrote: > > Ah. ok. The ssh issue isn't a big enough problem to warrant major code > changes in grossd. We've got the TIME_WAIT cleanup set to 60 seconds, > so we may try turning it down to 30 or 45 to see if it helps. Eventually I'm going to implement connection reusing in check_blocker. I'm still pondering if it should use asychronous I/O or not. If not, there should be means to clean up hung connections somehow. -- Eino Tuominen From jesse.thompson at doit.wisc.edu Wed Oct 31 22:57:55 2007 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Wed, 31 Oct 2007 15:57:55 -0500 Subject: [Gross] perpetual match Message-ID: <4728EC53.8040901@doit.wisc.edu> I generate a report from my gross logs that lists the top 20 IPs in the "Match" category, in an attempt to see which spammers are smart enough to retry and also find legitimate mailers that are being greylisted. I've noticed for some time that a couple of campus mail servers are always on the top of the match list. I ignored it for a long time figuring that they were just on a blacklist, and since they weren't complaining, no bother fixing it. I got curious one day and tried to figure out which blacklist they were on. They weren't listed on any blacklist that I query or the Sophos blocker. hmmm Next, I checked the gross logs. There were 0 Grey entries for these IP addresses going back 30 days. hmmm Why would this occur? My best guess is that there are false positives in the bloom filters. I have filter_bits = 24, so maybe I should raise it a bit to see if the problem goes away? But why does it only happen to these few mailers? Surely this would be more random? One thing that is unique about these servers is that the from address is relatively constant: server 1 (forwarded mail has the env_from rewritten): joeuser at deptartment.wisc.edu joeuser at wisc.edu or juser at deptartment.wisc.edu joeuser at wisc.edu server 2: mailman-bounces at department.wisc.edu joeuser at wisc.edu server 3: list-name-bounces at department.wisc.edu joeuser at wisc.edu Could this be leading to the bloom filter false positives? Anyway, this isn't really a big problem, just a minor annoyance since it screws up my stats. Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Tue Nov 6 22:42:08 2007 From: eino at utu.fi (Eino Tuominen) Date: Tue, 06 Nov 2007 22:42:08 +0200 Subject: [Gross] perpetual match In-Reply-To: <4728EC53.8040901@doit.wisc.edu> References: <4728EC53.8040901@doit.wisc.edu> Message-ID: <4730D1A0.6040100@utu.fi> Jesse Thompson wrote: > > Next, I checked the gross logs. There were 0 Grey entries for these IP > addresses going back 30 days. hmmm Check that you do have update = grey in config (it's the default). > But why does it only happen to these few mailers? Surely this would be > more random? It really is possible to have this kind of deterministic behaviour with bloom filters. Suppose you have a client that is constantly on some blocklist, and who regurarly sends email to the same recipient in your organization. The triplet is always the same and it always is present in your bloom filter. Suppose that these other triplets collide with the earlier triplet. The result is just what you are seeing. > Could this be leading to the bloom filter false positives? I'm afraid so. > Anyway, this isn't really a big problem, just a minor annoyance since it > screws up my stats. Yup. Spammers tend to be more random than real people, so this really is no big deal, I think. Sorry I didn't have time to answer you sooner, I've been pretty tied up elsewhere. -- From jesse.thompson at doit.wisc.edu Wed Nov 7 14:33:08 2007 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Wed, 07 Nov 2007 06:33:08 -0600 Subject: [Gross] perpetual match In-Reply-To: <4730D1A0.6040100@utu.fi> References: <4728EC53.8040901@doit.wisc.edu> <4730D1A0.6040100@utu.fi> Message-ID: <4731B084.40906@doit.wisc.edu> Eino Tuominen wrote: > Jesse Thompson wrote: >> >> Next, I checked the gross logs. There were 0 Grey entries for these >> IP addresses going back 30 days. hmmm > > Check that you do have update = grey in config (it's the default). It is >> But why does it only happen to these few mailers? Surely this would >> be more random? > > It really is possible to have this kind of deterministic behaviour with > bloom filters. Suppose you have a client that is constantly on some > blocklist, and who regurarly sends email to the same recipient in your > organization. The triplet is always the same and it always is present in > your bloom filter. Suppose that these other triplets collide with the > earlier triplet. The result is just what you are seeing. > >> Could this be leading to the bloom filter false positives? > > I'm afraid so. How likely will raising the filter bits setting help? Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From jesse.thompson at doit.wisc.edu Thu Nov 8 22:17:44 2007 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Thu, 08 Nov 2007 14:17:44 -0600 Subject: [Gross] perpetual match In-Reply-To: <4731B084.40906@doit.wisc.edu> References: <4728EC53.8040901@doit.wisc.edu> <4730D1A0.6040100@utu.fi> <4731B084.40906@doit.wisc.edu> Message-ID: <47336EE8.1070007@doit.wisc.edu> Jesse Thompson wrote: > Eino Tuominen wrote: >> Could this be leading to the bloom filter false positives? > I'm afraid so. Hmm. I see that I will need to reset the bloom filters to do this. Grossd shutdown with exit code 1: Configs differ! My: filter_size 25 number_buffers 12 Peer: filter_size 24 number_buffers 12 Is there any alternative? Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Fri Nov 9 00:11:55 2007 From: eino at utu.fi (Eino Tuominen) Date: Fri, 09 Nov 2007 00:11:55 +0200 Subject: [Gross] perpetual match In-Reply-To: <47336EE8.1070007@doit.wisc.edu> References: <4728EC53.8040901@doit.wisc.edu> <4730D1A0.6040100@utu.fi> <4731B084.40906@doit.wisc.edu> <47336EE8.1070007@doit.wisc.edu> Message-ID: <473389AB.7000603@utu.fi> Jesse Thompson wrote: > Jesse Thompson wrote: >> Eino Tuominen wrote: >>> Could this be leading to the bloom filter false positives? >> I'm afraid so. > > Hmm. I see that I will need to reset the bloom filters to do this. > > Grossd shutdown with exit code 1: Configs differ! > My: filter_size 25 number_buffers 12 > Peer: filter_size 24 number_buffers 12 > > Is there any alternative? Not at this moment. It requires modifications to the syncing protocol. There should be a force flag to disregard the configuration difference. Then you would be able to do this: 1) Prevent client queries to server A (change port or disable proto) 2) Resize filters and restart the server A 3) wait until the servers are in sync again (number_buffers times rotate_interval) 4) Make server A available to clients 5) do steps 1 to 4 with server B The force flag is fairly easy to implement. Look examples how to set up flags in gross.c, and then check the flag in syncmgr.c. Then, you must skip the initial sync (send_filters() function in syncmgr.c). -- Eino Tuominen From jesse.thompson at doit.wisc.edu Mon Nov 12 16:03:22 2007 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Mon, 12 Nov 2007 08:03:22 -0600 Subject: [Gross] perpetual match In-Reply-To: <47336EE8.1070007@doit.wisc.edu> References: <4728EC53.8040901@doit.wisc.edu> <4730D1A0.6040100@utu.fi> <4731B084.40906@doit.wisc.edu> <47336EE8.1070007@doit.wisc.edu> Message-ID: <47385D2A.5040403@doit.wisc.edu> Jesse Thompson wrote: > Jesse Thompson wrote: >> Eino Tuominen wrote: >>> Could this be leading to the bloom filter false positives? >> I'm afraid so. > > Hmm. I see that I will need to reset the bloom filters to do this. > > Grossd shutdown with exit code 1: Configs differ! > My: filter_size 25 number_buffers 12 > Peer: filter_size 24 number_buffers 12 > > Is there any alternative? After an initial stab at modifying the source code to ignore peer configuration differences, I decided that the impact of resetting the bloom filters was so benign that it wasn't worth adding more complexity to the code. I raised the filter_bits to 25 with a clean statefile late on Friday. No problems resulted from this. This seems to have solved the false positive issue for those campus MTAs that were always at the top of the list. Although, the match rate didn't seem to change (still at 4-6%,) which I was hoping would go lower. Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Mon Nov 12 17:48:36 2007 From: eino at utu.fi (Eino Tuominen) Date: Mon, 12 Nov 2007 17:48:36 +0200 Subject: [Gross] perpetual match In-Reply-To: <47385D2A.5040403@doit.wisc.edu> References: <4728EC53.8040901@doit.wisc.edu> <4730D1A0.6040100@utu.fi> <4731B084.40906@doit.wisc.edu> <47336EE8.1070007@doit.wisc.edu> <47385D2A.5040403@doit.wisc.edu> Message-ID: <473875D4.7030908@utu.fi> Jesse Thompson wrote: > > This seems to have solved the false positive issue for those campus MTAs > that were always at the top of the list. Although, the match rate > didn't seem to change (still at 4-6%,) which I was hoping would go lower. I just checked our statistics, we have seen 6.1% matches since September 13th. I haven't yet analyzed, if we see something similar. I rember that our match percentage was lower - could it be that spammers are starting to queue? I try to find time to see those logs more thoroughly. Now that SJSM offers Milter interface, it would be possible to demand that every message individually should be tried second time. That is, to calculate the hash from the whole message, not just the triplet. Of course that means more network bandwith would be wasted. -- Eino Tuominen From jesse.thompson at doit.wisc.edu Mon Nov 12 22:41:44 2007 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Mon, 12 Nov 2007 14:41:44 -0600 Subject: [Gross] perpetual match In-Reply-To: <473875D4.7030908@utu.fi> References: <4728EC53.8040901@doit.wisc.edu> <4730D1A0.6040100@utu.fi> <4731B084.40906@doit.wisc.edu> <47336EE8.1070007@doit.wisc.edu> <47385D2A.5040403@doit.wisc.edu> <473875D4.7030908@utu.fi> Message-ID: <4738BA88.7090604@doit.wisc.edu> Eino Tuominen wrote: > Jesse Thompson wrote: >> >> This seems to have solved the false positive issue for those campus MTAs >> that were always at the top of the list. Although, the match rate >> didn't seem to change (still at 4-6%,) which I was hoping would go lower. > > I just checked our statistics, we have seen 6.1% matches since September > 13th. I haven't yet analyzed, if we see something similar. I rember that > our match percentage was lower - could it be that spammers are starting > to queue? I try to find time to see those logs more thoroughly. Our match rate hasn't grown over time as much as you'd think. For whatever reason, the spammers generally aren't retrying (other than maybe firing off 2 messages simultaneously, which is prevented by the grey_delay=10 setting.) > Now that SJSM offers Milter interface, it would be possible to demand > that every message individually should be tried second time. That is, to > calculate the hash from the whole message, not just the triplet. Of > course that means more network bandwith would be wasted. That would just encourage them to retry, meanwhile wasting your resources. The ones that are retrying are probably properly retrying the same message anyway. I think the way to go is to raise the grey_delay to something higher as a first step. Then, if the spammers do retry, it gives the vendor/dnsbl spam traps time to catch up so that it makes your content scanning more effective. Plus, it would help if we build the sophistication into Gross to perm-block connections when the IP is listed on multiple (or weighted) blacklists. Is there a limit to how high you can raise the grey_delay setting? If I assume correctly, your implementation of this is just to keep the entry in memory for the grey_delay period of time before adding it to the bloom filter. I assume that it wouldn't be a good idea to raise the setting too high, right? As an alternative implementation, would it be possible to configure gross to query only the N-x oldest bloom filters? (N is the number of filters, x is the number of filters you want ignored) This way you achieve a large-value grey_delay behavior without the overhead of keeping the pending entries in memory. If this sounds like something that others would be interested in, I will take a stab at implementing it. Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From jesse.thompson at doit.wisc.edu Mon Nov 26 16:38:27 2007 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Mon, 26 Nov 2007 08:38:27 -0600 Subject: [Gross] thank you gross! Message-ID: <474ADA63.3010600@doit.wisc.edu> gross saved us from processing 100 million messages in one day! greylist: 102435214 (98.3%) 60610764 - 79.19.210.134 (host134-210-dynamic.19-79-r.retail.telecomitalia.it) 60610764 - @prudentialdoss.com 22485381 - 200.8.3.27 () 22485349 - @ibm.com 5 - @19.cn 5 - @stealthpost.com 5 - @telesensventures.com 5 - @sinagirl.com 5 - @4ur.com 4 - @clubadsl.fr 3 - @tvldyn.com 14804496 - 84.102.218.122 (122.218.102-84.rev.gaoland.net) 14804496 - @dc.rr.com -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Mon Nov 26 20:21:19 2007 From: eino at utu.fi (Eino Tuominen) Date: Mon, 26 Nov 2007 20:21:19 +0200 Subject: [Gross] thank you gross! In-Reply-To: <474ADA63.3010600@doit.wisc.edu> References: <474ADA63.3010600@doit.wisc.edu> Message-ID: <474B0E9F.409@utu.fi> Jesse Thompson wrote: > gross saved us from processing 100 million messages in one day! Wow! How did the system behave? Did you notice the attack while it was underway? -- Eino Tuominen From jesse.thompson at doit.wisc.edu Mon Nov 26 20:37:50 2007 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Mon, 26 Nov 2007 12:37:50 -0600 Subject: [Gross] thank you gross! In-Reply-To: <474B0E9F.409@utu.fi> References: <474ADA63.3010600@doit.wisc.edu> <474B0E9F.409@utu.fi> Message-ID: <474B127E.5030103@doit.wisc.edu> Eino Tuominen wrote: > Jesse Thompson wrote: >> gross saved us from processing 100 million messages in one day! > > Wow! How did the system behave? Did you notice the attack while it was > underway? We didn't notice anything at the time. My colleague actually mentioned that legitimate mail volumes looked low (assuming pre-holiday usage) because he saw the percentage in the 'trust' category drop, without noticing the overall gross load spike. It's hard to tell from the orca graphs if there was any impact on the servers, since we have gross running on sparse zones on 2 of our mail servers, and we don't have orca monitoring individual zones. The orca graphs for the those global zones show nothing out of the ordinary. The only thing noticeable was a spike in TCP Resets on all of the MTAs (including the ones that aren't running gross) I only imagine what would have happened if we actually had to process all of those messages! Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: