From eino at utu.fi Thu Feb 28 21:11:34 2008 From: eino at utu.fi (Eino Tuominen) Date: Thu, 28 Feb 2008 21:11:34 +0200 Subject: [Gross] Spammers adapt Message-ID: <47C70766.2000202@utu.fi> Hi, I just noticed that our match rate is much higher now than it has been before. I did some log analysis and found lots of patterns like this: Feb 28 19:39:32 smtp01 grossd: [ID 702911 mail.info] #a78: greylist: 190.48.141.76 jr.tolerd at kecoindustries.com our-user Feb 28 19:39:36 smtp01 grossd: [ID 702911 mail.info] #a79: greylist: 190.48.141.76 jr.tolerd at kecoindustries.com our-user Feb 28 19:39:47 smtp01 grossd: [ID 702911 mail.info] #a24: match: 190.48.141.76 jr.tolerd at kecoindustries.com our-user That is, retries go now beyond 10 seconds delay I've been using. I just resized our filter_bits to 26 and set grey_delay=120. Next I've decided to implement weights on checks and implement that block_threshold so grossd can reject obvious spammers with permanent errors. I've been pondering this kind of setup: dnsbl = bl.spamcop.net;2 dnsbl = dnsbl.njabl.org;1 dnsbl = dnsbl.sorbs.net;1 dnsbl = zen.spamhaus.org;3 dnsbl = rbl-plus.mail-abuse.org;3 block_threshold = 4 That is, no one match will lead to a permanent rejection, but two will be enough if one of them is highly trusted (spamhaus or rbl-plus). -- Eino Tuominen From R.E.Sonneveld at sonnection.nl Thu Feb 28 21:50:37 2008 From: R.E.Sonneveld at sonnection.nl (Rolf E. Sonneveld) Date: Thu, 28 Feb 2008 20:50:37 +0100 Subject: [Gross] Spammers adapt In-Reply-To: <47C70766.2000202@utu.fi> References: <47C70766.2000202@utu.fi> Message-ID: <47C7108D.4050204@sonnection.nl> Hi, Eino, Eino Tuominen wrote: > Hi, > > I just noticed that our match rate is much higher now than it has been > before. I did some log analysis and found lots of patterns like this: > > Feb 28 19:39:32 smtp01 grossd: [ID 702911 mail.info] #a78: greylist: > 190.48.141.76 jr.tolerd at kecoindustries.com our-user > Feb 28 19:39:36 smtp01 grossd: [ID 702911 mail.info] #a79: greylist: > 190.48.141.76 jr.tolerd at kecoindustries.com our-user > Feb 28 19:39:47 smtp01 grossd: [ID 702911 mail.info] #a24: match: > 190.48.141.76 jr.tolerd at kecoindustries.com our-user > > That is, retries go now beyond 10 seconds delay I've been using. I just > resized our filter_bits to 26 and set grey_delay=120. > > Next I've decided to implement weights on checks and implement that > block_threshold so grossd can reject obvious spammers with permanent > errors. I've been pondering this kind of setup: > > dnsbl = bl.spamcop.net;2 > dnsbl = dnsbl.njabl.org;1 > dnsbl = dnsbl.sorbs.net;1 > dnsbl = zen.spamhaus.org;3 > dnsbl = rbl-plus.mail-abuse.org;3 > block_threshold = 4 > in which version is this 'weight' factor available? /rolf From jesse.thompson at doit.wisc.edu Thu Feb 28 22:25:27 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Thu, 28 Feb 2008 14:25:27 -0600 Subject: [Gross] Spammers adapt In-Reply-To: <47C70766.2000202@utu.fi> References: <47C70766.2000202@utu.fi> Message-ID: <47C718B7.3090806@doit.wisc.edu> Eino Tuominen wrote: > Hi, > > I just noticed that our match rate is much higher now than it has been > before. I did some log analysis and found lots of patterns like this: Hmm. Ours is still around 3-4%. Overall volumes are up though. > Feb 28 19:39:32 smtp01 grossd: [ID 702911 mail.info] #a78: greylist: > 190.48.141.76 jr.tolerd at kecoindustries.com our-user > Feb 28 19:39:36 smtp01 grossd: [ID 702911 mail.info] #a79: greylist: > 190.48.141.76 jr.tolerd at kecoindustries.com our-user > Feb 28 19:39:47 smtp01 grossd: [ID 702911 mail.info] #a24: match: > 190.48.141.76 jr.tolerd at kecoindustries.com our-user Here is an idea that you might want to try. You can customize the SJSMS mapping response to include the $D flag which will cause the MTA to wait X milliseconds before responding. sjsms_response_grey = $X4.4.3|$N$D1000|Please$ try$ again$ later sjsms_response_match = $D$Y1000 We haven't tried this technique in production. > That is, retries go now beyond 10 seconds delay I've been using. I just > resized our filter_bits to 26 and set grey_delay=120. we still have grey_delay=10 we raised filter_bits to 25 a while ago after experiencing significant false positives in the bloom filters. Now, we're starting to see that again, so we might need to raise it again. One additional note, the bloom filter false positives tend to occur when there is a lot of repetition in the from/to values. You should doublecheck that your match percentage is not actually due to these false positives. > Next I've decided to implement weights on checks and implement that > block_threshold so grossd can reject obvious spammers with permanent > errors. I've been pondering this kind of setup: > > dnsbl = bl.spamcop.net;2 > dnsbl = dnsbl.njabl.org;1 > dnsbl = dnsbl.sorbs.net;1 > dnsbl = zen.spamhaus.org;3 > dnsbl = rbl-plus.mail-abuse.org;3 > block_threshold = 4 > > That is, no one match will lead to a permanent rejection, but two will > be enough if one of them is highly trusted (spamhaus or rbl-plus). That would be awesome. I think we discussed similar ideas here: http://code.google.com/p/gross/issues/detail?id=42 Don't forget to allow for a weight to be assigned to sophos blocker matches. Jesse -- Jesse Thompson Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Thu Feb 28 23:17:35 2008 From: eino at utu.fi (Eino Tuominen) Date: Thu, 28 Feb 2008 23:17:35 +0200 Subject: [Gross] Spammers adapt In-Reply-To: <47C718B7.3090806@doit.wisc.edu> References: <47C70766.2000202@utu.fi> <47C718B7.3090806@doit.wisc.edu> Message-ID: <47C724EF.5090705@utu.fi> Jesse Thompson wrote: > > sjsms_response_grey = $X4.4.3|$N$D1000|Please$ try$ again$ later > sjsms_response_match = $D$Y1000 > > We haven't tried this technique in production. I had some problems with that before. Though I think it was to do with mailing list expansion done at the reception time. I had to configure iMS to expand all mailing lists at the backend because of it. I have to check old conversations on iMS list about it... > we raised filter_bits to 25 a while ago after experiencing significant > false positives in the bloom filters. Now, we're starting to see that > again, so we might need to raise it again. I remembered that, and I decided to go straight up to 26 as it's still only a 100 megabytes of memory. > One additional note, the bloom filter false positives tend to occur when > there is a lot of repetition in the from/to values. You should > doublecheck that your match percentage is not actually due to these > false positives. I'll run a more thorough analysis tomorrow. > That would be awesome. > > I think we discussed similar ideas here: > http://code.google.com/p/gross/issues/detail?id=42 Yup, that's just the same block_threshold I'm talking about. > Don't forget to allow for a weight to be assigned to sophos blocker > matches. I'll get back to this tomorrow. -- Eino Tuominen From eino at utu.fi Thu Feb 28 23:18:53 2008 From: eino at utu.fi (Eino Tuominen) Date: Thu, 28 Feb 2008 23:18:53 +0200 Subject: [Gross] Spammers adapt In-Reply-To: <47C7108D.4050204@sonnection.nl> References: <47C70766.2000202@utu.fi> <47C7108D.4050204@sonnection.nl> Message-ID: <47C7253D.1090503@utu.fi> Rolf E. Sonneveld wrote: > > in which version is this 'weight' factor available? It's not yet available. I'll try to find some time to implement new things and ultimately release 1.0 not in too distant future. -- Eino Tuominen From eino at utu.fi Fri Feb 29 07:01:00 2008 From: eino at utu.fi (Eino Tuominen) Date: Fri, 29 Feb 2008 07:01:00 +0200 Subject: [Gross] Spammers adapt In-Reply-To: <47C70766.2000202@utu.fi> References: <47C70766.2000202@utu.fi> Message-ID: <47C7918C.3020609@utu.fi> Eino Tuominen wrote: > Hi, > > I just noticed that our match rate is much higher now than it has been > before. I did some log analysis and found lots of patterns like this: Stats from the last night: 8.9% trust 1.6% match 89.5% grey I suspect there are now some retries from spammers which raises grey percentage to such a high number. On the other hand, our spam volume keeps increasing, it's way over 90% of incoming mail. I'll write a script that analyzes log files and tells: 1) for every match, how many greylistings there were before it was accepted, and 2) time difference between the first greylisting and the match. -- Eino Tuominen From cadmoitaly at netscape.net Thu Mar 6 20:36:36 2008 From: cadmoitaly at netscape.net (cadmoitaly at netscape.net) Date: Thu, 06 Mar 2008 13:36:36 -0500 Subject: [Gross] Binaries and info on SJSMS Message-ID: <8CA4DD024E0C26B-15E0-55C@webmail-mf06.sysops.aol.com> 1. Do you have Solaris 10 x86 and SPARC ti be used with SJS Messaging Server 6.0 or 6.3? 2. Do you know how to set SJS Messaging Server 6.0 or 6.3 in order to use tcpd with related /ect/hosts.deny (twist myprog.sh) funcions? Thank you for publishing a really great work. Unfortunately I am not sure I succeed to test it due to compile! Regards from Marco Filippi cadmoitaly at netscape.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From teemu.toivanen at profnetti.fi Mon Mar 17 19:21:38 2008 From: teemu.toivanen at profnetti.fi (Teemu Toivanen) Date: Mon, 17 Mar 2008 19:21:38 +0200 Subject: [Gross] bug fix for HEAD (r212) #54 Message-ID: <1205774498.6697.26.camel@moon.profnetti.fi> There is a malloc/Free problem in HEAD. I created an issue #54 (http://code.google.com/p/gross/issues/detail?id=54) which covers the discovery process. In short, if there was a timeout in test_tuple and ta was created in test_tuple, the loop would change ta pointer to next with the result that later in code Free tries to free NULL pointer. Cheers, Teemu -------------- next part -------------- A non-text attachment was scrubbed... Name: gross_fix_worker_memoryleak_r212.patch Type: text/x-patch Size: 794 bytes Desc: not available URL: From eino at utu.fi Tue Mar 18 08:40:17 2008 From: eino at utu.fi (Eino Tuominen) Date: Tue, 18 Mar 2008 08:40:17 +0200 Subject: [Gross] bug fix for HEAD (r212) #54 In-Reply-To: <1205774498.6697.26.camel@moon.profnetti.fi> References: <1205774498.6697.26.camel@moon.profnetti.fi> Message-ID: <47DF63D1.3080607@utu.fi> Teemu Toivanen wrote: > There is a malloc/Free problem in HEAD. > I created an issue #54 > (http://code.google.com/p/gross/issues/detail?id=54) > > which covers the discovery process. > In short, if there was a timeout in test_tuple and ta was created in > test_tuple, the loop would change ta pointer to next with the result > that later in code Free tries to free NULL pointer. Thanks for the fix! It seems to be correct, too. With SJSMS ta (timeout action list) is supplied by the calling routine and the default timeout list is not created. That is why the bug has not come to our attention before. I'll commit it as soon as possible. -- Eino Tuominen From eino at utu.fi Wed Mar 19 23:27:14 2008 From: eino at utu.fi (Eino Tuominen) Date: Wed, 19 Mar 2008 23:27:14 +0200 Subject: [Gross] ANNOUNCE: new snapshot available (was: Re: bug fix for HEAD (r212) #54) In-Reply-To: <47DF63D1.3080607@utu.fi> References: <1205774498.6697.26.camel@moon.profnetti.fi> <47DF63D1.3080607@utu.fi> Message-ID: <47E18532.3070905@utu.fi> Eino Tuominen wrote: > That is why the bug has not come to our attention > before. I'll commit it as soon as possible. Fyi, I committed the patch and made a new snapshot available for download. I'm having some fun with our new F5 BigIP load balancers and upgrading our mail system, but I really want to find some time for coding real soon. I already have something done regarding dnsbl scoring and spf checks, but nothing ready to commit to svn, yet. Just hold your breath... ;-) -- Eino Tuominen From eino at utu.fi Sat Mar 22 23:11:34 2008 From: eino at utu.fi (Eino Tuominen) Date: Sat, 22 Mar 2008 23:11:34 +0200 Subject: [Gross] Wrong grey_mask default value Message-ID: <47E57606.2040308@utu.fi> Hello, I just noticed that the default value of grey_mask has been 0. It effectively means, that every client_ip is recorded as 0.0.0.0 in the filters. Do you have grey_mask set to some other value than default? That just might explain why I see more matches in the logs now than in early days. I have not yet decided what the default should be, I'm thinking setting it to 24 would be a safe bet. Also, weighted checks are coming, just having some weird segfaults keeping me from committing the changes to the svn trunk. Some time ago someone told me that 1111 was a bad choice for default port, indeed, there might be a situation when that port is not available for gross. I'm thinking ports 5225 for grossd, 5226 for sync and 5222 for stats. Feedback? Well, now back to debugging, -- Eino Tuominen From eino at utu.fi Mon Mar 24 10:49:13 2008 From: eino at utu.fi (Eino Tuominen) Date: Mon, 24 Mar 2008 10:49:13 +0200 Subject: [Gross] New features in svn trunk Message-ID: <47E76B09.1030807@utu.fi> Hi, Now there's weighted checks implemented in svn trunk. Only dnsbl and rhsbl weights are configurable so far, sophos blocker's weight is always 1, the default. Relevant configuration options are: block_threshold = [weightlimit] block_reason = reason # default: "Bad reputation" and dnsbl = dnslb[;weight] Example: block_threshold = 3 dnsbl = zen.spamhaus.org;3 dnsbl = bl.spamcop.net;2 dnsbl = combined.njabl.org dnsbl = dnsbl.sorbs.net With that configuration, gross will block if client_ip is found on zen.spamhaus.org alone, or in spamcop and some other list. Which lists do you consider more trustworthy? -- Eino Tuominen From jesse.thompson at doit.wisc.edu Mon Mar 24 20:52:01 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Mon, 24 Mar 2008 13:52:01 -0500 Subject: [Gross] Wrong grey_mask default value In-Reply-To: <47E57606.2040308@utu.fi> References: <47E57606.2040308@utu.fi> Message-ID: <47E7F851.4070108@doit.wisc.edu> Eino Tuominen wrote: > Hello, > > I just noticed that the default value of grey_mask has been 0. It > effectively means, that every client_ip is recorded as 0.0.0.0 in the > filters. Do you have grey_mask set to some other value than default? > That just might explain why I see more matches in the logs now than in > early days. I have not yet decided what the default should be, I'm > thinking setting it to 24 would be a safe bet. Mine isn't set. Are you sure? That's not the behavior I'm seeing. ~> /opt/wiscmail/gross/bin/gclient sjsms lasfjd laskjf 127.0.0.4 1 1111 got: G $X4.4.3|$NPlease$ try$ again$ later G $X4.4.3|$NPlease$ try$ again$ later ~> /opt/wiscmail/gross/bin/gclient sjsms lasfjd laskjf 127.0.0.4 1 1111 got: M $Y M $Y ~> /opt/wiscmail/gross/bin/gclient sjsms lasfjd laskjf 127.0.0.5 1 1111 got: G $X4.4.3|$NPlease$ try$ again$ later G $X4.4.3|$NPlease$ try$ again$ later > Also, weighted checks are coming, just having some weird segfaults > keeping me from committing the changes to the svn trunk. Cool! > Some time ago someone told me that 1111 was a bad choice for default > port, indeed, there might be a situation when that port is not available > for gross. I'm thinking ports 5225 for grossd, 5226 for sync and 5222 > for stats. 5222 is used by XMPP. Jesse -- Jesse Thompson Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Mon Mar 24 22:17:31 2008 From: eino at utu.fi (Eino Tuominen) Date: Mon, 24 Mar 2008 22:17:31 +0200 Subject: [Gross] Wrong grey_mask default value In-Reply-To: <47E7F851.4070108@doit.wisc.edu> References: <47E57606.2040308@utu.fi> <47E7F851.4070108@doit.wisc.edu> Message-ID: <47E80C5B.1060802@utu.fi> Jesse Thompson wrote: > Eino Tuominen wrote: >> Hello, >> >> I just noticed that the default value of grey_mask has been 0. It >> effectively means, that every client_ip is recorded as 0.0.0.0 in the >> filters. Do you have grey_mask set to some other value than default? >> That just might explain why I see more matches in the logs now than in >> early days. I have not yet decided what the default should be, I'm >> thinking setting it to 24 would be a safe bet. > > Mine isn't set. Are you sure? That's not the behavior I'm seeing. Well, I'd have to check svn logs when that's changed. What version do you run? > 5222 is used by XMPP. Ok, I'll try to google a better alternative. -- Eino Tuominen From jesse.thompson at doit.wisc.edu Mon Mar 24 23:05:34 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Mon, 24 Mar 2008 16:05:34 -0500 Subject: [Gross] Wrong grey_mask default value In-Reply-To: <47E80C5B.1060802@utu.fi> References: <47E57606.2040308@utu.fi> <47E7F851.4070108@doit.wisc.edu> <47E80C5B.1060802@utu.fi> Message-ID: <47E8179E.4050107@doit.wisc.edu> Eino Tuominen wrote: > Jesse Thompson wrote: >> Eino Tuominen wrote: >>> Hello, >>> >>> I just noticed that the default value of grey_mask has been 0. It >>> effectively means, that every client_ip is recorded as 0.0.0.0 in the >>> filters. Do you have grey_mask set to some other value than default? >>> That just might explain why I see more matches in the logs now than in >>> early days. I have not yet decided what the default should be, I'm >>> thinking setting it to 24 would be a safe bet. >> Mine isn't set. Are you sure? That's not the behavior I'm seeing. > > Well, I'd have to check svn logs when that's changed. What version do > you run? 0.8.2 > >> 5222 is used by XMPP. > > Ok, I'll try to google a better alternative. > -- Jesse Thompson Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Thu Mar 27 18:35:32 2008 From: eino at utu.fi (Eino Tuominen) Date: Thu, 27 Mar 2008 18:35:32 +0200 Subject: [Gross] Weighted checks in production Message-ID: <47EBCCD4.8010402@utu.fi> Hi all, I have now weighted checks version in production in one node: smtp01:gross# ps -e -o etime,time,vsz,rss,pcpu,args | grep grossd 03:50:08 6:08 32408 31776 0.6 sbin/grossd Trust: 9959 Match: 46 Greylist: 3643 Block: 39232 Queries/sec: 3.823019 Its now 6:30 pm here, so ham is still coming in. I uploaded a new development snapshot on Google, play with it! Oh, I now made these ports as defaults: 5525 gross; 5522 status; 5524 sync. This time I checked IANA port list first!-) -- Eino Tuominen From eino at utu.fi Thu Mar 27 18:57:11 2008 From: eino at utu.fi (Eino Tuominen) Date: Thu, 27 Mar 2008 18:57:11 +0200 Subject: [Gross] Weighted checks in production In-Reply-To: <47EBCCD4.8010402@utu.fi> References: <47EBCCD4.8010402@utu.fi> Message-ID: <47EBD1E7.4000408@utu.fi> Eino Tuominen wrote: > Hi all, > > I have now weighted checks version in production in one node: > > smtp01:gross# ps -e -o etime,time,vsz,rss,pcpu,args | grep grossd > 03:50:08 6:08 32408 31776 0.6 sbin/grossd > > Trust: 9959 Match: 46 Greylist: 3643 Block: 39232 Queries/sec: 3.823019 > > Its now 6:30 pm here, so ham is still coming in. I uploaded a new > development snapshot on Google, play with it! > > Oh, I now made these ports as defaults: 5525 gross; 5522 status; 5524 > sync. This time I checked IANA port list first!-) Darn, I managed to upload wrong version. Now check_blocker is broken in the gross-svn-232.tar.gz. I'll fix it ASAP. Otherwise it should be stable. I'll be back, -- Eino Tuominen From eino at utu.fi Fri Mar 28 12:43:20 2008 From: eino at utu.fi (Eino Tuominen) Date: Fri, 28 Mar 2008 12:43:20 +0200 Subject: [Gross] Weighted checks in production In-Reply-To: <47EBD1E7.4000408@utu.fi> References: <47EBCCD4.8010402@utu.fi> <47EBD1E7.4000408@utu.fi> Message-ID: <47ECCBC8.1090901@utu.fi> Eino Tuominen wrote: > Eino Tuominen wrote: >> Hi all, >> >> I have now weighted checks version in production in one node: >> >> smtp01:gross# ps -e -o etime,time,vsz,rss,pcpu,args | grep grossd >> 03:50:08 6:08 32408 31776 0.6 sbin/grossd >> >> Trust: 9959 Match: 46 Greylist: 3643 Block: 39232 Queries/sec: 3.823019 >> >> Its now 6:30 pm here, so ham is still coming in. I uploaded a new >> development snapshot on Google, play with it! >> >> Oh, I now made these ports as defaults: 5525 gross; 5522 status; 5524 >> sync. This time I checked IANA port list first!-) > > Darn, I managed to upload wrong version. Now check_blocker is broken in > the gross-svn-232.tar.gz. I'll fix it ASAP. Otherwise it should be stable. > > I'll be back, First night has passed, here are the statistics of one node: Trust: 15166 Match: 233 Greylist: 8582 Block: 120463 Quite amazing?! I have now uploaded gross-svn-236.tar.gz, it's running now in production here. I couldn't get check_blocker to reuse connections as Sophos' blockerd keeps shutting every connection down right after the answer. And, now there's watchdog to see if blocker threads get stuck. Jesse, you threw an idea some time ago about skipping first n rotating filters when calculating the effective filter. I've been pondering your idea and I think it could be of use now. It might be a good idea to hold a triplet as long as few hours at greylisted state, as it would give Puremessage and alike time to update their filters and let the spammers id get spread to another RBL's as well. What do you think? The original problem was that resolution of filters is so poor (1 hour with default config). But now that most spam gets blocked and only a portion gets greylisted there is a lot of room in the filters. We could easily afford more rotating buffers skipping a few first of them. -- Eino Tuominen From jesse.thompson at doit.wisc.edu Fri Mar 28 16:50:05 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Fri, 28 Mar 2008 09:50:05 -0500 Subject: [Gross] Weighted checks in production In-Reply-To: <47ECCBC8.1090901@utu.fi> References: <47EBCCD4.8010402@utu.fi> <47EBD1E7.4000408@utu.fi> <47ECCBC8.1090901@utu.fi> Message-ID: <47ED059D.3040603@doit.wisc.edu> Eino Tuominen wrote: > Eino Tuominen wrote: >> Eino Tuominen wrote: >>> Hi all, >>> >>> I have now weighted checks version in production in one node: >>> >>> smtp01:gross# ps -e -o etime,time,vsz,rss,pcpu,args | grep grossd >>> 03:50:08 6:08 32408 31776 0.6 sbin/grossd >>> >>> Trust: 9959 Match: 46 Greylist: 3643 Block: 39232 Queries/sec: 3.823019 >>> >>> Its now 6:30 pm here, so ham is still coming in. I uploaded a new >>> development snapshot on Google, play with it! >>> >>> Oh, I now made these ports as defaults: 5525 gross; 5522 status; 5524 >>> sync. This time I checked IANA port list first!-) >> Darn, I managed to upload wrong version. Now check_blocker is broken in >> the gross-svn-232.tar.gz. I'll fix it ASAP. Otherwise it should be stable. >> >> I'll be back, > > First night has passed, here are the statistics of one node: > > Trust: 15166 Match: 233 Greylist: 8582 Block: 120463 > > Quite amazing?! Cool! Nice work, as always. I will try this out soon, I hope. > I have now uploaded gross-svn-236.tar.gz, it's running now in production > here. I couldn't get check_blocker to reuse connections as Sophos' > blockerd keeps shutting every connection down right after the answer. > And, now there's watchdog to see if blocker threads get stuck. Was this a bug? I can't remember the details. > Jesse, you threw an idea some time ago about skipping first n rotating > filters when calculating the effective filter. I've been pondering your > idea and I think it could be of use now. It might be a good idea to hold > a triplet as long as few hours at greylisted state, as it would give > Puremessage and alike time to update their filters and let the spammers > id get spread to another RBL's as well. What do you think? Yeah, I think that this is the most efficient way to enable longer greylisted states. From what I understand, the current method of setting a greylist time period keeps the entry in memory and waits X seconds before updating the filter. This seems to me like it wouldn't scale well to larger time values. > The original problem was that resolution of filters is so poor (1 hour > with default config). But now that most spam gets blocked and only a > portion gets greylisted there is a lot of room in the filters. We could > easily afford more rotating buffers skipping a few first of them. Yeah, if the filter resolution is 1 hour, and you tell gross to ignore the first filter, it will mean that the triplets will be in the greylist state for 0 < X < 1 hour. Otherwise you could tell gross to ignore the first 2 filters, which would be 1 hour < X < 2 hours. That seems too long. I'd probably want to first try setting the filter resolution to 30 minutes and then ignore the first 2 filters, so the triplets will be in the greylist state for 30 min < X < 1 hour. That still might be a little too aggressive though. So maybe 15 minute filters, ignoring the first 2: 15 min < X < 30 min? Thoughts? Jesse -- Jesse Thompson Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From jesse.thompson at doit.wisc.edu Mon Mar 31 19:43:34 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Mon, 31 Mar 2008 11:43:34 -0500 Subject: [Gross] ips.backscatterer.org Message-ID: <47F114B6.1070104@doit.wisc.edu> There are some backscatter/outscatter storms going on, and while our mail servers aren't having any problems handling the load, some of our users are being swamped. There is a DNSBL called backscatterer.org that attempts to list all mail servers that have sent backscatter or have done sender callouts in the past week. http://www.backscatterer.org/ Has anyone tried querying this DNSBL with gross? Does it have any effect (for better or worse?) Does it make a difference that the envelope from address is null? Jesse -- Jesse Thompson Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From jesse.thompson at doit.wisc.edu Tue Apr 1 23:00:40 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Tue, 01 Apr 2008 15:00:40 -0500 Subject: [Gross] ips.backscatterer.org In-Reply-To: <47F114B6.1070104@doit.wisc.edu> References: <47F114B6.1070104@doit.wisc.edu> Message-ID: <47F29468.1090004@doit.wisc.edu> meh. looks like the senders that are listed just retry, so there isn't much point in greylisting backscatter unless you see this as a form of punishment, or contribution to a weighted threshold for blocking. > grep "<>" /var/log/grosslog | grep -c trust 3348 > grep "<>" /var/log/grosslog | grep -c match 982 > grep "<>" /var/log/grosslog | grep -c grey 856 0: Grossd OK. Update queue: 1488 (In: 1488 + Out: 0) Trust: 303364 Match: 51545 Greylist: 844001 Queries/sec: 83.623492 Dnsbl matches: grossd dnsbl matches (bl.spamcop.net, dnsbl.njabl.org, cbl.abuseat.org, dnsbl.sorbs.net, list.dsbl.org, multihop.dsbl.org, ips.backscatterer.org): 463784, 9178, 188087, 150695, 678, 71, 28933 (note, to make sense of these stats, only one mta (of 4) was sending DSNs to gross, and we have 2 gross servers. the stats reflect 4 hours of traffic) Jesse Jesse Thompson wrote: > There are some backscatter/outscatter storms going on, and while our > mail servers aren't having any problems handling the load, some of our > users are being swamped. > > There is a DNSBL called backscatterer.org that attempts to list all mail > servers that have sent backscatter or have done sender callouts in the > past week. > http://www.backscatterer.org/ > > Has anyone tried querying this DNSBL with gross? Does it have any > effect (for better or worse?) Does it make a difference that the > envelope from address is null? > > Jesse > > > ------------------------------------------------------------------------ > > _______________________________________________ > Gross mailing list > Gross at lists.utu.fi > https://lists.utu.fi/mailman/listinfo/gross -- Jesse Thompson Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Tue Apr 1 23:28:38 2008 From: eino at utu.fi (Eino Tuominen) Date: Tue, 01 Apr 2008 23:28:38 +0300 Subject: [Gross] ips.backscatterer.org In-Reply-To: <47F29468.1090004@doit.wisc.edu> References: <47F114B6.1070104@doit.wisc.edu> <47F29468.1090004@doit.wisc.edu> Message-ID: <47F29AF6.4000500@utu.fi> Jesse Thompson wrote: > meh. looks like the senders that are listed just retry, so there isn't > much point in greylisting backscatter unless you see this as a form of That's what I expected. After all, to be a backscatterer you have to run an MTA. > 0: Grossd OK. Update queue: 1488 (In: 1488 + Out: 0) Trust: 303364 > Match: 51545 Greylist: 844001 Queries/sec: 83.623492 Dnsbl matches: > grossd dnsbl matches (bl.spamcop.net, dnsbl.njabl.org, cbl.abuseat.org, > dnsbl.sorbs.net, list.dsbl.org, multihop.dsbl.org, > ips.backscatterer.org): 463784, 9178, 188087, 150695, 678, 71, 28933 > > (note, to make sense of these stats, only one mta (of 4) was sending > DSNs to gross, and we have 2 gross servers. the stats reflect 4 hours > of traffic) Your amount of traffic always amazes me... So, you get over 160 recipients per second during peak hours? How many PMX hosts do you run? And how many would you run if you didn't run Gross? -- Eino Tuominen From jesse.thompson at doit.wisc.edu Tue Apr 1 23:40:09 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Tue, 01 Apr 2008 15:40:09 -0500 Subject: [Gross] ips.backscatterer.org In-Reply-To: <47F29AF6.4000500@utu.fi> References: <47F114B6.1070104@doit.wisc.edu> <47F29468.1090004@doit.wisc.edu> <47F29AF6.4000500@utu.fi> Message-ID: <47F29DA9.4000709@doit.wisc.edu> Eino Tuominen wrote: > Jesse Thompson wrote: >> meh. looks like the senders that are listed just retry, so there >> isn't much point in greylisting backscatter unless you see this as a >> form of > > That's what I expected. After all, to be a backscatterer you have to run > an MTA. > >> 0: Grossd OK. Update queue: 1488 (In: 1488 + Out: 0) Trust: 303364 >> Match: 51545 Greylist: 844001 Queries/sec: 83.623492 Dnsbl matches: >> grossd dnsbl matches (bl.spamcop.net, dnsbl.njabl.org, cbl.abuseat.org, >> dnsbl.sorbs.net, list.dsbl.org, multihop.dsbl.org, >> ips.backscatterer.org): 463784, 9178, 188087, 150695, 678, 71, 28933 >> >> (note, to make sense of these stats, only one mta (of 4) was sending >> DSNs to gross, and we have 2 gross servers. the stats reflect 4 hours >> of traffic) > > Your amount of traffic always amazes me... So, you get over 160 > recipients per second during peak hours? How many PMX hosts do you run? > And how many would you run if you didn't run Gross? We have 8 dedicated pmx servers at the moment (4 Dell 2650, 4 Dell 1950) but that's scaled to pre-gross levels (10 million messages a day). With gross, we only have to deal with around 2 million messages a day. We could probably scale back to 3-4 1950s if we wanted to push it (or get better value from our hardware). The rest of our email servers are Sun Sparc Solaris 10, so we're considering switching pmx to run on zones on Sparc T5220 sparse zones. This would allow us to dual purpose machines that are otherwise underutilized. Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From jesse.thompson at doit.wisc.edu Wed Apr 2 19:09:12 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Wed, 02 Apr 2008 11:09:12 -0500 Subject: [Gross] uceprotect.net Message-ID: <47F3AFA8.30201@doit.wisc.edu> Any thoughts on querying UCEPROTECT with gross? I configured one of our gross servers to query dnsbl-2.uceprotect.net. The trust rate on this server dropped to 13%, while the other is at 27%. Would it be too BOFHish to try dnsbl-3.uceprotect.net? :-) Jesse -- Jesse Thompson Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Wed Apr 2 19:41:59 2008 From: eino at utu.fi (Eino Tuominen) Date: Wed, 02 Apr 2008 19:41:59 +0300 Subject: [Gross] uceprotect.net In-Reply-To: <47F3AFA8.30201@doit.wisc.edu> References: <47F3AFA8.30201@doit.wisc.edu> Message-ID: <47F3B757.90003@utu.fi> Jesse Thompson wrote: > Any thoughts on querying UCEPROTECT with gross? I configured one of our > gross servers to query dnsbl-2.uceprotect.net. The trust rate on this > server dropped to 13%, while the other is at 27%. > > Would it be too BOFHish to try dnsbl-3.uceprotect.net? :-) I have concidered to try it, but have not done it yet. Sounds like a good candidate to use with gross. I have trust: 15.4 match: 0.6 greylist: 7.2 block: 78.5 zen.spamhaus seems to be a really efficient list, too. I have now: grey_delay=120 check = dnsbl check = rhsbl check = blocker blocker_host = pmx.utu.fi dnsbl=bl.spamcop.net dnsbl=dnsbl.njabl.org dnsbl=dnsbl.sorbs.net rhsbl=rhsbl.sorbs.net rhsbl=rddn.dnsbl.net.au dnsbl=zen.spamhaus.org;3 dnsbl=rbl-plus.mail-abuse.org;3 block_threshold = 4 Both of those rhsbl lists seem to be of no use. -- Eino Tuominen From jesse.thompson at doit.wisc.edu Wed Apr 2 20:06:06 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Wed, 02 Apr 2008 12:06:06 -0500 Subject: [Gross] uceprotect.net In-Reply-To: <47F3B757.90003@utu.fi> References: <47F3AFA8.30201@doit.wisc.edu> <47F3B757.90003@utu.fi> Message-ID: <47F3BCFE.2070103@doit.wisc.edu> Eino Tuominen wrote: > zen.spamhaus seems to be a really efficient list, too. I have now: Are you paying them and rsyncing? I disabled this dnsbl since they want large sites to pay. > dnsbl=bl.spamcop.net > dnsbl=dnsbl.njabl.org > dnsbl=dnsbl.sorbs.net We use those too. > rhsbl=rhsbl.sorbs.net > rhsbl=rddn.dnsbl.net.au > Both of those rhsbl lists seem to be of no use. too bad :-( > dnsbl=rbl-plus.mail-abuse.org;3 I'll have to try rbl-plus.mail-abuse.org. We use cbl.abuseat.org too. It seems to catch a lot. Jesse -- Jesse Thompson Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Wed Apr 2 20:16:15 2008 From: eino at utu.fi (Eino Tuominen) Date: Wed, 02 Apr 2008 20:16:15 +0300 Subject: [Gross] uceprotect.net In-Reply-To: <47F3BCFE.2070103@doit.wisc.edu> References: <47F3AFA8.30201@doit.wisc.edu> <47F3B757.90003@utu.fi> <47F3BCFE.2070103@doit.wisc.edu> Message-ID: <47F3BF5F.4090303@utu.fi> Jesse Thompson wrote: > Eino Tuominen wrote: >> zen.spamhaus seems to be a really efficient list, too. I have now: > > Are you paying them and rsyncing? I disabled this dnsbl since they want > large sites to pay. Yep. It's a few thousand bucks a year for academic organizations. See http://www.spamteq.com/ma/datafeed/pricecalculator.lasso > I'll have to try rbl-plus.mail-abuse.org. It's commercial too. (Trend Micro nowadays). -- Eino Tuominen From eino at utu.fi Thu Apr 3 21:28:02 2008 From: eino at utu.fi (Eino Tuominen) Date: Thu, 03 Apr 2008 21:28:02 +0300 Subject: [Gross] Binaries and info on SJSMS In-Reply-To: <8CA4DD024E0C26B-15E0-55C@webmail-mf06.sysops.aol.com> References: <8CA4DD024E0C26B-15E0-55C@webmail-mf06.sysops.aol.com> Message-ID: <47F521B2.7000202@utu.fi> cadmoitaly at netscape.net wrote: > 1. Do you have Solaris 10 x86 and SPARC ti be used with SJS Messaging > Server 6.0 or 6.3? Hello, Sorry for the late response. No, we do not have binaries. But gross is very easy to compile, basically it's just ./configure && make. You can download Sun Studio compiling tools from http://www.sun.com, it's free of charge. Or, you can use gcc if you prefer. We might produce binaries after 1.0 release. > 2. Do you know how to set SJS Messaging Server 6.0 or 6.3 in order to > use tcpd with related /ect/hosts.deny (twist myprog.sh) funcions? > Thank you for publishing a really great work. Unfortunately I am not > sure I succeed to test it due to compile! I'm sorry, but I don't understand your question. Do you mean that you would like to use tcp_wrappers in front of SJSMS? IMHO, there's no need for it, you can configure access control in SJSMS. -- Eino Tuominen From eino at utu.fi Fri Apr 4 15:25:35 2008 From: eino at utu.fi (Eino Tuominen) Date: Fri, 04 Apr 2008 15:25:35 +0300 Subject: [Gross] Binaries and info on SJSMS In-Reply-To: <8CA64626947D7D4-13C8-3F4A@WEBMAIL-DC20.sysops.aol.com> References: <8CA4DD024E0C26B-15E0-55C@webmail-mf06.sysops.aol.com> <47F521B2.7000202@utu.fi> <8CA64626947D7D4-13C8-3F4A@WEBMAIL-DC20.sysops.aol.com> Message-ID: <47F61E3F.3080003@utu.fi> cadmoitaly at netscape.net wrote: > THANK YOU FOR REPLY! > > 1. I'll try, but I have a few hopes... Bad relationship with cc. As I said, compiling Gross on Solaris is pretty straight forward task. > So, do you know how to set SJSMS in order to use tcpd? Better place to ask this is info-ims at arnold.com -mailing list. But as far as I know, it is not possible to run dispatcher (SMTP server of SJSMS suite) with tcp_wrapper. But dynamic (as you mean it) rules are certainly possible to implement. You can use general database and configure PORT_ACCESS mapping to query to use it. Or, you can do same thing with ldap. Or you could write a specific library to do that. Or, you could add those evil IP:s to your own private dnsbl file (google rbldnsd) and have grossd to block all connections from hosts that match the dnsbl. -- Eino Tuominen From eino at utu.fi Fri Apr 4 16:59:51 2008 From: eino at utu.fi (Eino Tuominen) Date: Fri, 04 Apr 2008 16:59:51 +0300 Subject: [Gross] Binaries and info on SJSMS In-Reply-To: <8CA6469FAE420EC-13C8-41D3@WEBMAIL-DC20.sysops.aol.com> References: <8CA4DD024E0C26B-15E0-55C@webmail-mf06.sysops.aol.com> <47F521B2.7000202@utu.fi> <8CA64626947D7D4-13C8-3F4A@WEBMAIL-DC20.sysops.aol.com> <47F61DDE.3000502@utu.fi> <8CA6469FAE420EC-13C8-41D3@WEBMAIL-DC20.sysops.aol.com> Message-ID: <47F63457.2010401@utu.fi> cadmoitaly at netscape.net wrote: > > Afaik, a dnsbl file used by gross gives a positive reply for matching > ip, so that ths positive reply is used to deny an access. In the current downloadable version you can use also dns white lists. Basically, you can configure a weight for the check. If you want to block matching IPs completely, you configure weight of the dnsbl to be equal to or greater than block_threshold. > Moreover, more important and rilevant, PORT_ACCESS sections map IPs to > be allowed or denied. *Is there a way to twist / run a simple.sh script?* > Thank you for your kind reply, I urge you to ask SJSMS specific questions on SJSMS mailing lists or web forums. I think that only way to run a shell script from mappings would be to write a site supplied library that runs the script and evaluates it's return values. I understand that you are not very familiar with C programming, so that might not be realistic approach for you, though. -- Eino Tuominen From jesse.thompson at doit.wisc.edu Fri Apr 4 17:14:51 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Fri, 04 Apr 2008 09:14:51 -0500 Subject: [Gross] Binaries and info on SJSMS In-Reply-To: <47F63457.2010401@utu.fi> References: <8CA4DD024E0C26B-15E0-55C@webmail-mf06.sysops.aol.com> <47F521B2.7000202@utu.fi> <8CA64626947D7D4-13C8-3F4A@WEBMAIL-DC20.sysops.aol.com> <47F61DDE.3000502@utu.fi> <8CA6469FAE420EC-13C8-41D3@WEBMAIL-DC20.sysops.aol.com> <47F63457.2010401@utu.fi> Message-ID: <47F637DB.3030900@doit.wisc.edu> Eino Tuominen wrote: >> Moreover, more important and rilevant, PORT_ACCESS sections map IPs to >> be allowed or denied. *Is there a way to twist / run a simple.sh script?* >> Thank you for your kind reply, > > I urge you to ask SJSMS specific questions on SJSMS mailing lists or web > forums. Yes, ask info-ims. > I think that only way to run a shell script from mappings would be to > write a site supplied library that runs the script and evaluates it's > return values. I understand that you are not very familiar with C > programming, so that might not be realistic approach for you, though. You can use the CONVERSIONS channel to run scripts. Chad's site has some (old) examples. http://ims.balius.com/resources/downloads/index.html Read the docs. They've added lots of new features recently, and I haven't been able to research in depth yet. -- Jesse Thompson Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Sat Apr 5 09:33:07 2008 From: eino at utu.fi (Eino Tuominen) Date: Sat, 05 Apr 2008 09:33:07 +0300 Subject: [Gross] Weighted checks in production In-Reply-To: <47ED059D.3040603@doit.wisc.edu> References: <47EBCCD4.8010402@utu.fi> <47EBD1E7.4000408@utu.fi> <47ECCBC8.1090901@utu.fi> <47ED059D.3040603@doit.wisc.edu> Message-ID: <47F71D23.5020407@utu.fi> Jesse Thompson wrote: > > Cool! Nice work, as always. I will try this out soon, I hope. It would be nice to get some feedback how the current version is behaving. >> I have now uploaded gross-svn-236.tar.gz, it's running now in >> production here. I couldn't get check_blocker to reuse connections as >> Sophos' blockerd keeps shutting every connection down right after the >> answer. And, now there's watchdog to see if blocker threads get stuck. > > Was this a bug? I can't remember the details. I don't remember if there has been any discussion about this in perl-mx mailing list. But I even configured a test server to have postfix make policy queries to blockerd, and even then it's one query / connection. Blockerd just closes the connection. I will do some reverse engineering with sendmail. It would be really nice if those connections could be reused. > Yeah, I think that this is the most efficient way to enable longer > greylisted states. From what I understand, the current method of > setting a greylist time period keeps the entry in memory and waits X > seconds before updating the filter. This seems to me like it wouldn't > scale well to larger time values. Yes, the current implementation uses a delayed message queue implemented in msgqueue.c. Message queues are basically linked lists with mutexes, and delay queue is just two message queues (inq and outq) with a handler thread moving messages from inq to outq. It actually scales well performance-wise. The biggest problem with them is that they are saved in memory. Should both replicated grossd nodes fail at the same time, the queue is lost. Now that most of the messages will be blocked and only a fraction will be greylisted, it would be possible to use a lot bigger number_buffers with a bit smaller filter_bits and rotate_interval. How about 22 bits, 15 minutes and 48 buffers? PID COMMAND RPRVT RSHRD RSIZE VSIZE 13410 grossd 25M 200K- 25M 46M New options to implement could be: 'skip_buffers' = how many (maximum) buffers to skip 'skip_style' = how to do the skipping, 'absolute' means that grossd always puts update on the first buffer, 'weighed' (or something) would place more suspicious triplets in the first buffer, and less suspicious triplets nearer to skip limit, so that the greylisting time would be relative to susp_weight. If you set skip_buffers = 16 (around 3 hours), skip_style = weighed and block_threshold = 9, grey_threshold = 1, then grossd would calculate the filter to update like this s = how many to skip, S = skip_buffers, w = total weight from the checks, b = block_threshold, g = grey_threshold s = S * ( w - g ) / ( b - g - 1) With the given config this would be s = 16 * ( w - 1 ) / ( 9 - 1 - 1) = 16 / 7 * ( w - 1) If w = 1, then s = 0, with w = 8, s = 16 With that, more suspicious hosts would wait more, and less suspicious would have to wait less. Thoughts? This is not entirely trivial modification, as it means modifications in the code that has been very stable since early days. The good thing is that this is easy to stresstest if implemented. -- Eino Tuominen From eino at utu.fi Sat Apr 5 09:37:49 2008 From: eino at utu.fi (Eino Tuominen) Date: Sat, 05 Apr 2008 09:37:49 +0300 Subject: [Gross] uceprotect.net In-Reply-To: <47F3AFA8.30201@doit.wisc.edu> References: <47F3AFA8.30201@doit.wisc.edu> Message-ID: <47F71E3D.7090506@utu.fi> Jesse Thompson wrote: > Any thoughts on querying UCEPROTECT with gross? I configured one of our > gross servers to query dnsbl-2.uceprotect.net. The trust rate on this > server dropped to 13%, while the other is at 27%. > > Would it be too BOFHish to try dnsbl-3.uceprotect.net? :-) In one node I have had uceprotect.net (ndsbl-3 included :-) for a day now. Here are the results: blocker_weight = 8 dnsbl=bl.spamcop.net;3 dnsbl=dnsbl.njabl.org;3 dnsbl=dnsbl.sorbs.net;3 dnsbl=dnsbl-1.uceprotect.net;3 dnsbl=dnsbl-2.uceprotect.net;1 dnsbl=dnsbl-3.uceprotect.net;1 dnsbl=rbl-plus.mail-abuse.org;6 dnsbl=zen.spamhaus.org;6 dnswl=dnswl.org block_threshold = 9 trust: 12.4 match: 0.2 greylist: 8.4 block: 79 And compared to the other node: blocker_host = pmx.utu.fi dnsbl=bl.spamcop.net dnsbl=dnsbl.njabl.org dnsbl=dnsbl.sorbs.net rhsbl=rhsbl.sorbs.net dnsbl=zen.spamhaus.org;3 dnsbl=rbl-plus.mail-abuse.org;3 block_threshold = 4 trust: 14 match: 0.2 greylist: 6.8 block: 79.1 -- Eino Tuominen From eino at utu.fi Mon Apr 7 23:39:15 2008 From: eino at utu.fi (Eino Tuominen) Date: Mon, 07 Apr 2008 23:39:15 +0300 Subject: [Gross] Announce: Gross 0.9 Message-ID: <47FA8673.80608@utu.fi> Hi, There's now 0.9 release available for download. It seemed that release names like svn-r123 were too scary... -- Eino Tuominen From jesse.thompson at doit.wisc.edu Tue Apr 8 01:25:20 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Mon, 07 Apr 2008 17:25:20 -0500 Subject: [Gross] Announce: Gross 0.9 In-Reply-To: <47FA8673.80608@utu.fi> References: <47FA8673.80608@utu.fi> Message-ID: <47FA9F50.90507@doit.wisc.edu> Yes, this makes it easier for me to coincide log history in our configuration cvs repository. 0.8 < svn-r123 < 0.9 That will throw me off a year from now when its not fresh in my mind. Also, people don't like 0.x (not stable) or x.0 (will wait for first maintenance release) I'll deploy 0.9 soon so that you can justify releasing 1.0 Jesse Eino Tuominen wrote: > Hi, > > There's now 0.9 release available for download. It seemed that release > names like svn-r123 were too scary... > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Wed Apr 9 20:50:56 2008 From: eino at utu.fi (Eino Tuominen) Date: Wed, 09 Apr 2008 20:50:56 +0300 Subject: [Gross] Released Gross 0.9.1 Message-ID: <47FD0200.7050604@utu.fi> Hi, I had to release 0.9.1 already, as --disable-dnsbl configure option was broken. See details of issue #55 at googlecode. 0.9 and 0.9.1 are identical if compiled without --disable-dnsbl. -- Eino Tuominen From jesse.thompson at doit.wisc.edu Wed Apr 9 20:53:32 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Wed, 09 Apr 2008 12:53:32 -0500 Subject: [Gross] Weighted checks in production In-Reply-To: <47F71D23.5020407@utu.fi> References: <47EBCCD4.8010402@utu.fi> <47EBD1E7.4000408@utu.fi> <47ECCBC8.1090901@utu.fi> <47ED059D.3040603@doit.wisc.edu> <47F71D23.5020407@utu.fi> Message-ID: <47FD029C.608@doit.wisc.edu> Eino Tuominen wrote: > Jesse Thompson wrote: >> >> Cool! Nice work, as always. I will try this out soon, I hope. > > It would be nice to get some feedback how the current version is behaving. > >>> I have now uploaded gross-svn-236.tar.gz, it's running now in >>> production here. I couldn't get check_blocker to reuse connections as >>> Sophos' blockerd keeps shutting every connection down right after the >>> answer. And, now there's watchdog to see if blocker threads get stuck. >> >> Was this a bug? I can't remember the details. > > I don't remember if there has been any discussion about this in perl-mx > mailing list. But I even configured a test server to have postfix make > policy queries to blockerd, and even then it's one query / connection. > Blockerd just closes the connection. I will do some reverse engineering > with sendmail. It would be really nice if those connections could be > reused. > >> Yeah, I think that this is the most efficient way to enable longer >> greylisted states. From what I understand, the current method of >> setting a greylist time period keeps the entry in memory and waits X >> seconds before updating the filter. This seems to me like it wouldn't >> scale well to larger time values. > > Yes, the current implementation uses a delayed message queue implemented > in msgqueue.c. Message queues are basically linked lists with mutexes, > and delay queue is just two message queues (inq and outq) with a handler > thread moving messages from inq to outq. It actually scales well > performance-wise. The biggest problem with them is that they are saved > in memory. Should both replicated grossd nodes fail at the same time, > the queue is lost. > > Now that most of the messages will be blocked and only a fraction will > be greylisted, it would be possible to use a lot bigger number_buffers > with a bit smaller filter_bits and rotate_interval. How about 22 bits, > 15 minutes and 48 buffers? > > PID COMMAND RPRVT RSHRD RSIZE VSIZE > 13410 grossd 25M 200K- 25M 46M > > New options to implement could be: > > 'skip_buffers' = how many (maximum) buffers to skip > > 'skip_style' = how to do the skipping, 'absolute' means that grossd > always puts update on the first buffer, 'weighed' (or something) would > place more suspicious triplets in the first buffer, and less suspicious > triplets nearer to skip limit, so that the greylisting time would be > relative to susp_weight. > > If you set skip_buffers = 16 (around 3 hours), skip_style = weighed > and block_threshold = 9, grey_threshold = 1, then grossd would calculate > the filter to update like this > > s = how many to skip, S = skip_buffers, w = total weight from the > checks, b = block_threshold, g = grey_threshold > > s = S * ( w - g ) / ( b - g - 1) > > With the given config this would be > > s = 16 * ( w - 1 ) / ( 9 - 1 - 1) = 16 / 7 * ( w - 1) > > If w = 1, then s = 0, with w = 8, s = 16 > > With that, more suspicious hosts would wait more, and less suspicious > would have to wait less. > > Thoughts? This is not entirely trivial modification, as it means > modifications in the code that has been very stable since early days. > The good thing is that this is easy to stresstest if implemented. I think it's a cool idea. It might be overkill though. Would it be easier to just start with the 'skip_buffers' option with the absolute 'skip_style' behavior? Jesse -- Jesse Thompson Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Fri Apr 11 11:13:16 2008 From: eino at utu.fi (Eino Tuominen) Date: Fri, 11 Apr 2008 11:13:16 +0300 Subject: [Gross] Gross 0.9.2 out Message-ID: <47FF1D9C.2010608@utu.fi> Hi, I wrapped up release 0.9.2, which fixes a few bugs reported to me. If things go smoothly at Wisconsin, next release will be 1.0. -- Eino Tuominen From jesse.thompson at doit.wisc.edu Sat Apr 12 01:32:53 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Fri, 11 Apr 2008 17:32:53 -0500 Subject: [Gross] gross FAQ Message-ID: <47FFE715.2070307@doit.wisc.edu> Here is a first version of a FAQ for gross. Feel free to correct my mistakes, reorder (I didn't put any thought into the ordering) and add whatever you think is missing. http://code.google.com/p/gross/wiki/FAQ Jesse -- Jesse Thompson Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Sat Apr 12 07:57:39 2008 From: eino at utu.fi (Eino Tuominen) Date: Sat, 12 Apr 2008 07:57:39 +0300 Subject: [Gross] gross FAQ In-Reply-To: <47FFE715.2070307@doit.wisc.edu> References: <47FFE715.2070307@doit.wisc.edu> Message-ID: <48004143.7000306@utu.fi> Jesse Thompson wrote: > Here is a first version of a FAQ for gross. Feel free to correct my > mistakes, reorder (I didn't put any thought into the ordering) and add > whatever you think is missing. > > http://code.google.com/p/gross/wiki/FAQ Looking good. It might be a good idea to make own sections for sjsms and sophos specific questions. Also, I elaborated bloom filter answer a bit. Thanks! -- Eino Tuominen From eino at utu.fi Tue Apr 15 13:10:59 2008 From: eino at utu.fi (Eino Tuominen) Date: Tue, 15 Apr 2008 13:10:59 +0300 Subject: [Gross] Ready for 1.0? Message-ID: <48047F33.80102@utu.fi> Hi, Does anyone know any standing issues which might prevent me from releasing 1.0? Performance is of no concern anymore, as if I'm not mistaken, 0.9.2 is running at Wisc without any problems or performance issues, even when running only with one node. -- Eino Tuominen From naked at iki.fi Tue Apr 15 14:20:19 2008 From: naked at iki.fi (Nuutti Kotivuori) Date: Tue, 15 Apr 2008 14:20:19 +0300 Subject: [Gross] New to the list Message-ID: <87mynvl5zg.fsf@aka.i.naked.iki.fi> Hello, I'm new to Gross and new to the list, so just announcing myself here. I intend to run Gross on a few very small installations as the only spam blocker - just to cut off the majority of needless spam. I also would love to see a Debian package out of Gross - but possibly will not have the time to go through the channels myself. I will probably post some small patches here which will create a rudimentary debian package - but nothing fit for inclusion in a distribution. I'm posting a couple comments on the software itself in separate mails here, so they do not get jumbled up with the rest of the message. Gross seems like a fine piece of software - keep up the good work! -- Naked From naked at iki.fi Tue Apr 15 14:26:19 2008 From: naked at iki.fi (Nuutti Kotivuori) Date: Tue, 15 Apr 2008 14:26:19 +0300 Subject: [Gross] Makefile cleanup for installing grossd.conf Message-ID: <87iqyjl5pg.fsf@aka.i.naked.iki.fi> Hi, I found this bit in the Makefile.am: install-data-local: if test ! -f $(sysconfdir)/grossd.conf; then $(INSTALL_DATA) $(srcdir)/doc/examples/grossd.conf $(sysconfdir); fi As a minor point, I dislike the test for existence of grossd.conf. Sure, that's handy when testing and developing, but those sorts of things in the Makefile make things hard for the package maintainers - and can confuse some users. But the real problem is that the commandline doesn't take in to account $(DESTDIR). It should be: install-data-local: if test ! -f $(DESTDIR)$(sysconfdir)/grossd.conf; then $(INSTALL_DATA) $(srcdir)/doc/examples/grossd.conf $(DESTDIR)$(sysconfdir); fi Or just the same without the test for existence before. Without this fix, installing with DESTDIR causes probably a permission error from the system install location - and in any case the configuration file lands up outside the directory specified. -- Naked From naked at iki.fi Tue Apr 15 14:32:31 2008 From: naked at iki.fi (Nuutti Kotivuori) Date: Tue, 15 Apr 2008 14:32:31 +0300 Subject: [Gross] Pidfile and daemonization Message-ID: <87bq4bl5f4.fsf@aka.i.naked.iki.fi> Hi, I'm having a spot of bother with using a pidfile with grossd. As far as I can see, there's no option in grossd itself for creating a pidfile. Since grossd forks and daemonizes itself by default (and does it nicely after configuring grossd so configuration errors can be caught), it would be prudent for it to automatically create a pidfile if so requested. This can ofcourse be worked around by not having the daemon fork and creating the pidfile externally. However, it seems that grossd disables logging to syslog in this case - which prevents this from being used in system installations. So, I'd like to request either pidfile creation - or the ability to not-daemonize and still log to syslog. I might come up with a patch for this if necessary, when I can muster the time. -- Naked From Dejan.Krnjevic at saga.co.yu Tue Apr 15 14:33:31 2008 From: Dejan.Krnjevic at saga.co.yu (Dejan Krnjevic) Date: Tue, 15 Apr 2008 13:33:31 +0200 Subject: [Gross] Gross milter Sendmail Message-ID: Hi, I am trying to install Gross with Sendmail milter. Does anyone knows what's sintax for milter sock in sendmail.mc file? I have searched trough readmes and didn't find any tip about it. Anyone can help me? -------------- next part -------------- An HTML attachment was scrubbed... URL: From eino at utu.fi Tue Apr 15 14:56:39 2008 From: eino at utu.fi (Eino Tuominen) Date: Tue, 15 Apr 2008 14:56:39 +0300 Subject: [Gross] Gross milter Sendmail In-Reply-To: References: Message-ID: <480497F7.2020108@utu.fi> Dejan Krnjevic wrote: > Hi, > > > > I am trying to install Gross with Sendmail milter. Does anyone knows > what?s sintax for milter sock in sendmail.mc file? > > I have searched trough readmes and didn?t find any tip about it. Try adding this in sendmail.mc: INPUT_MAIL_FILTER(`Gross', `S=inet:5523 at localhost, T=R:20s') And of course in grossd.conf: protocol = milter milter_listen = inet:1234 at localhost Remember, the milter code in grossd has not been tested in a large environment. -- Eino Tuominen From eino at utu.fi Tue Apr 15 15:01:18 2008 From: eino at utu.fi (Eino Tuominen) Date: Tue, 15 Apr 2008 15:01:18 +0300 Subject: [Gross] Makefile cleanup for installing grossd.conf In-Reply-To: <87iqyjl5pg.fsf@aka.i.naked.iki.fi> References: <87iqyjl5pg.fsf@aka.i.naked.iki.fi> Message-ID: <4804990E.4000001@utu.fi> Nuutti Kotivuori wrote: > Hi, > > I found this bit in the Makefile.am: > > install-data-local: > if test ! -f $(sysconfdir)/grossd.conf; then $(INSTALL_DATA) $(srcdir)/doc/examples/grossd.conf $(sysconfdir); fi > > As a minor point, I dislike the test for existence of grossd.conf. > Sure, that's handy when testing and developing, but those sorts of > things in the Makefile make things hard for the package maintainers - > and can confuse some users. Can you give me an example, please. I have thought that overwriting existing config would make things even more confusing, when upgrading. > But the real problem is that the commandline doesn't take in to > account $(DESTDIR). > > It should be: > > install-data-local: > if test ! -f $(DESTDIR)$(sysconfdir)/grossd.conf; then $(INSTALL_DATA) $(srcdir)/doc/examples/grossd.conf $(DESTDIR)$(sysconfdir); fi > Thats a bug, thanks for pointing it out! I'll fix it shortly. -- Eino Tuominen From eino at utu.fi Tue Apr 15 15:04:03 2008 From: eino at utu.fi (Eino Tuominen) Date: Tue, 15 Apr 2008 15:04:03 +0300 Subject: [Gross] Pidfile and daemonization In-Reply-To: <87bq4bl5f4.fsf@aka.i.naked.iki.fi> References: <87bq4bl5f4.fsf@aka.i.naked.iki.fi> Message-ID: <480499B3.70901@utu.fi> Nuutti Kotivuori wrote: > > So, I'd like to request either pidfile creation - or the ability to > not-daemonize and still log to syslog. Creating a pidfile is easy to implement, and does not mess with the actual server code as it can be done very early at the startup phase. Please, make an RFE at http://code.google.com/p/gross/issues. -- Eino Tuominen From naked at iki.fi Tue Apr 15 15:31:39 2008 From: naked at iki.fi (Nuutti Kotivuori) Date: Tue, 15 Apr 2008 15:31:39 +0300 Subject: [Gross] Makefile cleanup for installing grossd.conf In-Reply-To: <4804990E.4000001@utu.fi> References: <87iqyjl5pg.fsf@aka.i.naked.iki.fi> <4804990E.4000001@utu.fi> Message-ID: <873apnl2ok.fsf@aka.i.naked.iki.fi> Eino Tuominen wrote: > Nuutti Kotivuori wrote: >> Hi, >> >> I found this bit in the Makefile.am: >> >> install-data-local: >> if test ! -f $(sysconfdir)/grossd.conf; then $(INSTALL_DATA) >> $(srcdir)/doc/examples/grossd.conf $(sysconfdir); fi >> >> As a minor point, I dislike the test for existence of grossd.conf. >> Sure, that's handy when testing and developing, but those sorts of >> things in the Makefile make things hard for the package maintainers - >> and can confuse some users. > > Can you give me an example, please. I have thought that overwriting > existing config would make things even more confusing, when upgrading. Well, the issue is not that the behaviour would be confusing in itself, it's just that it's different from the common norm on how most packages do it. As a user, I might expect the file to be overwritten on 'make install' (especially if my prefix is something else than /) and then wonder why nothing works as the configuration file is still from the old version - though I don't think I would ever 'upgrade' a piece of software just by compiling a new version and saying make install. But, like said, a minor point, perhaps relevant only to me. >> But the real problem is that the commandline doesn't take in to >> account $(DESTDIR). >> >> It should be: >> >> install-data-local: >> if test ! -f $(DESTDIR)$(sysconfdir)/grossd.conf; then $(INSTALL_DATA) >> $(srcdir)/doc/examples/grossd.conf $(DESTDIR)$(sysconfdir); fi > > Thats a bug, thanks for pointing it out! I'll fix it shortly. Actually, you could handle this the way most packages do it: confdir = $(sysconfdir) conf_DATA = grossd.conf That would require moving grossd.conf to the root of the source distribution though, if there are no changes in the source search paths. Obviously it's entirely up to you, either goes. -- Naked From naked at iki.fi Tue Apr 15 15:33:01 2008 From: naked at iki.fi (Nuutti Kotivuori) Date: Tue, 15 Apr 2008 15:33:01 +0300 Subject: [Gross] Pidfile and daemonization In-Reply-To: <480499B3.70901@utu.fi> References: <87bq4bl5f4.fsf@aka.i.naked.iki.fi> <480499B3.70901@utu.fi> Message-ID: <87y77fjo1u.fsf@aka.i.naked.iki.fi> Eino Tuominen wrote: > Nuutti Kotivuori wrote: >> >> So, I'd like to request either pidfile creation - or the ability to >> not-daemonize and still log to syslog. > > Creating a pidfile is easy to implement, and does not mess with the > actual server code as it can be done very early at the startup > phase. Please, make an RFE at http://code.google.com/p/gross/issues. That I did. However, I didn't find any options to flag it as an enchancement request (haven't used the google code issue tracker before), so I hope that can be fixed. -- Naked From teemu.toivanen at profnetti.fi Tue Apr 15 15:52:06 2008 From: teemu.toivanen at profnetti.fi (Teemu Toivanen) Date: Tue, 15 Apr 2008 15:52:06 +0300 Subject: [Gross] Makefile cleanup for installing grossd.conf In-Reply-To: <873apnl2ok.fsf@aka.i.naked.iki.fi> References: <87iqyjl5pg.fsf@aka.i.naked.iki.fi> <4804990E.4000001@utu.fi> <873apnl2ok.fsf@aka.i.naked.iki.fi> Message-ID: <1208263926.5228.39.camel@moon.profnetti.fi> I'm not a packager and lately I've been avoiding src only projects. That said here is my 2 cents worth. I would expect "make install" not to screw with my config files and therefore would never want them to be overwritten. I can accept that example config file is located in package doc and no default file is created (though for initial src install it's nice that it will be created). As for packagers the "package create environment" should always be clean and therefore there will be no problem with "old config". Cheers, Teemu On Tue, 2008-04-15 at 15:31 +0300, Nuutti Kotivuori wrote: > Eino Tuominen wrote: > > Nuutti Kotivuori wrote: > >> Hi, > >> > >> I found this bit in the Makefile.am: > >> > >> install-data-local: > >> if test ! -f $(sysconfdir)/grossd.conf; then $(INSTALL_DATA) > >> $(srcdir)/doc/examples/grossd.conf $(sysconfdir); fi > >> > >> As a minor point, I dislike the test for existence of grossd.conf. > >> Sure, that's handy when testing and developing, but those sorts of > >> things in the Makefile make things hard for the package maintainers - > >> and can confuse some users. > > > > Can you give me an example, please. I have thought that overwriting > > existing config would make things even more confusing, when upgrading. > > Well, the issue is not that the behaviour would be confusing in > itself, it's just that it's different from the common norm on how most > packages do it. > > As a user, I might expect the file to be overwritten on 'make install' > (especially if my prefix is something else than /) and then wonder why > nothing works as the configuration file is still from the old version > - though I don't think I would ever 'upgrade' a piece of software just > by compiling a new version and saying make install. > > But, like said, a minor point, perhaps relevant only to me. From eino at utu.fi Tue Apr 15 20:30:55 2008 From: eino at utu.fi (Eino Tuominen) Date: Tue, 15 Apr 2008 20:30:55 +0300 Subject: [Gross] Pidfile and daemonization In-Reply-To: <87bq4bl5f4.fsf@aka.i.naked.iki.fi> References: <87bq4bl5f4.fsf@aka.i.naked.iki.fi> Message-ID: <4804E64F.8030705@utu.fi> Nuutti Kotivuori wrote: > > So, I'd like to request either pidfile creation - or the ability to > not-daemonize and still log to syslog. As I've never been a fan of pidfiles I want to know exactly what do you expect grossd to do with pidfiles: 1. should grossd check if the pidfile already exists? 2. should grossd remove the pidfile before exiting? FWIW, I always use pgrep et al, it's more reliable than pidfiles, which could end up being stale. -- Eino Tuominen From jesse.thompson at doit.wisc.edu Tue Apr 15 20:41:48 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Tue, 15 Apr 2008 12:41:48 -0500 Subject: [Gross] Pidfile and daemonization In-Reply-To: <4804E64F.8030705@utu.fi> References: <87bq4bl5f4.fsf@aka.i.naked.iki.fi> <4804E64F.8030705@utu.fi> Message-ID: <4804E8DC.7000503@doit.wisc.edu> Eino Tuominen wrote: > Nuutti Kotivuori wrote: >> So, I'd like to request either pidfile creation - or the ability to >> not-daemonize and still log to syslog. > > As I've never been a fan of pidfiles I want to know exactly what do you > expect grossd to do with pidfiles: > > 1. should grossd check if the pidfile already exists? > 2. should grossd remove the pidfile before exiting? > > FWIW, I always use pgrep et al, it's more reliable than pidfiles, which > could end up being stale. We're using the Solaris Service Management Facility to start/stop grossd. SMF will prevent two grossd instances from being started at once, and it will automatically restart grossd if it fails. If anyone is interested, I will contribute our smf definition file to the project. Where should I put it? What should I call it? It would be nice to get some feedback for improvements, etc. Jesse -- Jesse Thompson Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From naked at iki.fi Tue Apr 15 21:07:07 2008 From: naked at iki.fi (Nuutti Kotivuori) Date: Tue, 15 Apr 2008 21:07:07 +0300 Subject: [Gross] Pidfile and daemonization In-Reply-To: <4804E64F.8030705@utu.fi> References: <87bq4bl5f4.fsf@aka.i.naked.iki.fi> <4804E64F.8030705@utu.fi> Message-ID: <87wsmzhu0k.fsf@aka.i.naked.iki.fi> Eino Tuominen wrote: > Nuutti Kotivuori wrote: >> So, I'd like to request either pidfile creation - or the ability to >> not-daemonize and still log to syslog. > > As I've never been a fan of pidfiles I want to know exactly what do > you expect grossd to do with pidfiles: > > 1. should grossd check if the pidfile already exists? > 2. should grossd remove the pidfile before exiting? > > FWIW, I always use pgrep et al, it's more reliable than pidfiles, > which could end up being stale. For me, a workable solution would be to not check that the pidfile already exists - that is, just overwrite it when starting. Removing the pidfile on exit would be nice, but not entirely necessary. I personally think that the wrapping scripts should handle all the "housekeeping" around pidfiles - and the daemon should just create the pidfile if it forks itself. But some others will most likely disagree. Many daemons do check the pidfile first and try to determine if the process is still alive. -- Naked From eino at utu.fi Tue Apr 15 21:37:16 2008 From: eino at utu.fi (Eino Tuominen) Date: Tue, 15 Apr 2008 21:37:16 +0300 Subject: [Gross] Pidfile and daemonization In-Reply-To: <4804E8DC.7000503@doit.wisc.edu> References: <87bq4bl5f4.fsf@aka.i.naked.iki.fi> <4804E64F.8030705@utu.fi> <4804E8DC.7000503@doit.wisc.edu> Message-ID: <4804F5DC.6030904@utu.fi> Jesse Thompson wrote: > > If anyone is interested, I will contribute our smf definition file to > the project. Where should I put it? What should I call it? It would > be nice to get some feedback for improvements, etc. It's already there, doc/examples/smf-definition.xml. We are yet to upgrade our MTA's onto Solaris 10, so I haven't use it myself. I think you should remove dependency to pmx, though. Grossd can handle unresponsive blocker (actually, blocker threadpool just fills up). -- Eino Tuominen From jesse.thompson at doit.wisc.edu Tue Apr 15 21:51:13 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Tue, 15 Apr 2008 13:51:13 -0500 Subject: [Gross] Pidfile and daemonization In-Reply-To: <4804F5DC.6030904@utu.fi> References: <87bq4bl5f4.fsf@aka.i.naked.iki.fi> <4804E64F.8030705@utu.fi> <4804E8DC.7000503@doit.wisc.edu> <4804F5DC.6030904@utu.fi> Message-ID: <4804F921.2090707@doit.wisc.edu> Eino Tuominen wrote: > Jesse Thompson wrote: >> >> If anyone is interested, I will contribute our smf definition file to >> the project. Where should I put it? What should I call it? It would >> be nice to get some feedback for improvements, etc. > > It's already there, doc/examples/smf-definition.xml. We are yet to > upgrade our MTA's onto Solaris 10, so I haven't use it myself. I think My memory is crap I guess. > you should remove dependency to pmx, though. Grossd can handle > unresponsive blocker (actually, blocker threadpool just fills up). I sort of agree with that, and I sort of don't. By that logic, you don't need a dependency on name services either. But, I wouldn't want a grossd node handling production traffic without either. Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Tue Apr 15 22:07:17 2008 From: eino at utu.fi (Eino Tuominen) Date: Tue, 15 Apr 2008 22:07:17 +0300 Subject: [Gross] Pidfile and daemonization In-Reply-To: <4804F921.2090707@doit.wisc.edu> References: <87bq4bl5f4.fsf@aka.i.naked.iki.fi> <4804E64F.8030705@utu.fi> <4804E8DC.7000503@doit.wisc.edu> <4804F5DC.6030904@utu.fi> <4804F921.2090707@doit.wisc.edu> Message-ID: <4804FCE5.2050808@utu.fi> Jesse Thompson wrote: > > I sort of agree with that, and I sort of don't. By that logic, you > don't need a dependency on name services either. But, I wouldn't want a > grossd node handling production traffic without either. You have a point in there. It would be best if smf-manifest could be used without modifications at most sites. I cannot grasp if you can make sort of 'weak' dependency checks, that is to depend on a service only if such service is enabled. -- Eino Tuominen From jesse.thompson at doit.wisc.edu Tue Apr 15 23:36:16 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Tue, 15 Apr 2008 15:36:16 -0500 Subject: [Gross] Pidfile and daemonization In-Reply-To: <4804FCE5.2050808@utu.fi> References: <87bq4bl5f4.fsf@aka.i.naked.iki.fi> <4804E64F.8030705@utu.fi> <4804E8DC.7000503@doit.wisc.edu> <4804F5DC.6030904@utu.fi> <4804F921.2090707@doit.wisc.edu> <4804FCE5.2050808@utu.fi> Message-ID: <480511C0.9040008@doit.wisc.edu> Eino Tuominen wrote: > Jesse Thompson wrote: >> >> I sort of agree with that, and I sort of don't. By that logic, you >> don't need a dependency on name services either. But, I wouldn't want >> a grossd node handling production traffic without either. > > You have a point in there. It would be best if smf-manifest could be > used without modifications at most sites. I cannot grasp if you can make > sort of 'weak' dependency checks, that is to depend on a service only if > such service is enabled. > OK, I just commented them out. It can be uncommented for those that want those dependencies. Jesse -- Jesse Thompson Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Thu Apr 17 23:06:53 2008 From: eino at utu.fi (Eino Tuominen) Date: Thu, 17 Apr 2008 23:06:53 +0300 Subject: [Gross] Gross 0.9.3 Message-ID: <4807ADDD.408@utu.fi> Hello, I just uploaded release 0.9.3 to the Google project site. It features new log format, and pidfile creation. I think it's now even closer to 1.0. I'm running this version here now, and haven't seen any problems with it. Standard disclaimers apply... ;-) Here is the NEWS entry: Version 0.9.3 * HELOified all the rest of the code. There are still no checks that use helo info, but now grossd logs it and also gclient can send it to the server. Check INSTALL details about configuring SJSMS to send helo string over to grossd. * New logfile format! An example log line: Thu Apr 17 14:50:05 2008 #8505600: a=greylist d=3 w=5 c=127.0.0.2 s=sender r=recipient h=utu.fi m=cbl.abuseat.org+1 m=bl.spamcop.net+2 m=combined.njabl.org+1 m=dnsbl.sorbs.net+1 where a = action d = delay w = total weight c = client ip s = sender r = recipient h = helo/ehlo m = matched check with possible weight - A few important bug fixes to milter server - behaviour change: grossd now exits after creating the statefile when invoked with -C command line option. This makes life easier for package maintainers in the future. - Implemented pidfile creation and checking. New config option 'pidfile' controls the behaviour. Setting the option will make grossd to write the process pid into the file specified by the full path. An optional parameter 'check' can be set to keep grossd from starting if the pidfile already exists. pidfile = /full/path[;check] Another way to control this is by command line options -p and -P. Both of the options will enable pidfile creation, and -P will check the existence of the file. Issues fixed: #58: grossd should write a pid file after daemonizing (RFE) -- Eino Tuominen From eino at utu.fi Mon Apr 21 20:19:30 2008 From: eino at utu.fi (Eino Tuominen) Date: Mon, 21 Apr 2008 20:19:30 +0300 Subject: [Gross] Gross 1.0rc out! Message-ID: <480CCCA2.7000604@utu.fi> Hello, I just uploaded release candidate for 1.0 release! There are yet a few new things included: - Manual pages. Yes, really, I wrote manual pages for grossd and grossd.conf. Please, read 'em through and report any mistakes or grammar errors or whatever. Even I make mistakes ;-) Especially you native English speakers! - Prevent grossd from running as root. This is a crucial change to test on different OS's. If started as root (geteuid() == 0) grossd will run setuid(getpwnam("nobody")) before even opening log file. (with error checks, of course). - Also, the installing with DESTDIR=/foo/bar should work now properly. When I get enough feedback I'll release 1.0.0. About the future: I just got spf check working, but I decided not to include it in 1.0. Thanks you all! Check out AUTHORS -file. If you feel you have entitled to be included in the credits, let me know. I tried to remember all contributions from the past, but some I might have forgotten. -- Eino Tuominen From eino at utu.fi Tue Apr 22 22:12:54 2008 From: eino at utu.fi (Eino Tuominen) Date: Tue, 22 Apr 2008 22:12:54 +0300 Subject: [Gross] Grossd 1.0rc2 Message-ID: <480E38B6.2060809@utu.fi> Hi, I'm spamming you yet again. There's a new release candidate available. You may want to use it instead of rc1 especially if you are going to review the man pages or wan't to be package maintainers. Changes: - Adds a trailing dot (if not already there) to all dns queries in order to make sure all queries will be made as fully qualified. - Passes make distcheck. -- Eino Tuominen From eino at utu.fi Wed Apr 23 00:07:41 2008 From: eino at utu.fi (Eino Tuominen) Date: Wed, 23 Apr 2008 00:07:41 +0300 Subject: [Gross] gross-1.0rc2 In-Reply-To: <480E3EE9.1040801@doit.wisc.edu> References: <480E3EE9.1040801@doit.wisc.edu> Message-ID: <480E539D.8050401@utu.fi> Jesse Thompson wrote: > why is grossd.conf now called /opt/gross/etc? I just can't get it right... I accidentally deleted a line from Makefile.am: sysconf_DATA = If $sysconfdir didn't already exist, make install happily copied grossd.conf as $PREFIX/etc. So, once again, a new release candidate is available... -- Eino Tuominen From jesse.thompson at doit.wisc.edu Wed Apr 23 23:53:20 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Wed, 23 Apr 2008 15:53:20 -0500 Subject: [Gross] Grossd 1.0rc2 In-Reply-To: <480E38B6.2060809@utu.fi> References: <480E38B6.2060809@utu.fi> Message-ID: <480FA1C0.6050907@doit.wisc.edu> Hi, I just upgraded to 1.0rc3 in production, and everything looks good so far. The man pages aren't formatting correctly for me though. Jesse Eino Tuominen wrote: > Hi, > > I'm spamming you yet again. There's a new release candidate available. > You may want to use it instead of rc1 especially if you are going to > review the man pages or wan't to be package maintainers. > > Changes: > - Adds a trailing dot (if not already there) to all dns queries in order > to make sure all queries will be made as fully qualified. > - Passes make distcheck. > -- Jesse Thompson Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Thu Apr 24 06:20:57 2008 From: eino at utu.fi (Eino Tuominen) Date: Thu, 24 Apr 2008 06:20:57 +0300 Subject: [Gross] Grossd 1.0rc2 In-Reply-To: <480FA1C0.6050907@doit.wisc.edu> References: <480E38B6.2060809@utu.fi> <480FA1C0.6050907@doit.wisc.edu> Message-ID: <480FFC99.4010508@utu.fi> Jesse Thompson wrote: > Hi, > > I just upgraded to 1.0rc3 in production, and everything looks good so > far. The man pages aren't formatting correctly for me though. It's because I wrote them using a macro package that wasn't portable enough. (doc) They should format on BSD and Linux. I'll reformat them using the good old man macros. It's PITA, but I have not found a better way. If someone knows, please tell me. -- Eino Tuominen From jff.chan at gmail.com Thu Apr 24 13:44:39 2008 From: jff.chan at gmail.com (Jeff Chan) Date: Thu, 24 Apr 2008 18:44:39 +0800 Subject: [Gross] Space Character in Mail From Crashing Gross Message-ID: <696c56870804240344x6bac796bmbf98230f2c88ef4f@mail.gmail.com> Hi, I started to experiment with Gross few days ago with SJSMS. I observed that Gross crashed regularly, without any error message, and thus left the server out of greylisting protection. I use Gross 1.0rc2 and found it crashses when a spammer sends something like this: MAIL FROM: SIZE=1000 The mail.log shows the MS takes spam"mer at dom> SIZE=1000 as the envelop address, without taking care of the unbalanced double quote. Although I think it's either a MS bug or at least a configuration issue, Gross shouldn't be crashing because of invalid inputs. Anyone encounters this issue? Or just me? And BTW, a little bit off topic, what's your experience dealing with invalid characters using the MS configuration files like mappings? Thanks. Jeff From jesse.thompson at doit.wisc.edu Thu Apr 24 15:57:56 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Thu, 24 Apr 2008 07:57:56 -0500 Subject: [Gross] Space Character in Mail From Crashing Gross In-Reply-To: <696c56870804240344x6bac796bmbf98230f2c88ef4f@mail.gmail.com> References: <696c56870804240344x6bac796bmbf98230f2c88ef4f@mail.gmail.com> Message-ID: <481083D4.4030104@doit.wisc.edu> Hi Jeff, I'll let Eino address the gross crashing issue. But I have a workaround for you. This FROM_ACCESS mapping will do the trick. After seeing a lot of garbage in our queues due to 'mail from's with control characters, we did an analysis (from our mail logs and searching around for what others are doing) on a reasonable policy for valid characters in the local part of the mail from. I know that we should technically be allowing more characters per the RFC, but we've been running like this for a while and haven't had any complaints. Jesse (in case you can't tell, the space character is covered by the range ascii hex 0x01-0x21 (space is 0x20) which shows up visually as ^A-! note: the ^A is what is printed in the terminal, but it's actually ascii hex 0x01. Let me know if you need help typing that in (it's not ^ and A); it's possible to do with vim and a bit of googling.) FROM_ACCESS ! ! Require EHLO/HELO *|*|*|*|*|SMTP/|*|*|*|* $N$D900|EHLO/HELO$ argument$ required ! ! Prohibit spaces in EHLO/HELO *|*|*|*|*|SMTP/*$ *|*|*|*|* $NSpace$ not$ allowed$ in$ EHLO/HELO ! ! Prohibit invalid characters in anywhere in the uid ! ascii hex: 01-21, 24, 25, 28, 29, 2c, 3a, 3b, 3c, 3e, 5b-5e, 60, 7b, 7d *|*|*|*|*|*|*|*|*$[^A-!$%(),;^`\{\}]%*@*|* $Ninvalid$ character$ in$ mail$ from ! ! Prohibit single character questionable characters ! in the local part of the envelope from address ! ascii hex: 23, 26, 27, 2a, 3b, 3f, 2b, 2d, 2e, 5f, 7e *|*|*|*|*|*|*|*|$[#&'*=?+\-._\~]%@*|* $Ninvalid$ character$ in$ mail$ from ! ! Prohibit quoted single character questionable characters ! in the local part of the envelope from address ! ascii hex: 23, 26, 27, 2a, 3b, 3f, 2b, 2d, 2e, 5f, 7e *|*|*|*|*|*|*|*|"$[#&'*=?+\-._\~]%"@*|* $Ninvalid$ character$ in$ mail$ from Jeff Chan wrote: > Hi, > > I started to experiment with Gross few days ago with SJSMS. I observed > that Gross crashed regularly, without any error message, and thus left > the server out of greylisting protection. > > I use Gross 1.0rc2 and found it crashses when a spammer sends > something like this: > > MAIL FROM: SIZE=1000 > > The mail.log shows the MS takes spam"mer at dom> SIZE=1000 as the envelop > address, without taking care of the unbalanced double quote. Although > I think it's either a MS bug or at least a configuration issue, Gross > shouldn't be crashing because of invalid inputs. > > Anyone encounters this issue? Or just me? And BTW, a little bit off > topic, what's your experience dealing with invalid characters using > the MS configuration files like mappings? > > Thanks. > > Jeff > _______________________________________________ > Gross mailing list > Gross at lists.utu.fi > https://lists.utu.fi/mailman/listinfo/gross -- Jesse Thompson Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From jff.chan at gmail.com Thu Apr 24 16:57:33 2008 From: jff.chan at gmail.com (Jeff Chan) Date: Thu, 24 Apr 2008 21:57:33 +0800 Subject: [Gross] Space Character in Mail From Crashing Gross In-Reply-To: <481083D4.4030104@doit.wisc.edu> References: <696c56870804240344x6bac796bmbf98230f2c88ef4f@mail.gmail.com> <481083D4.4030104@doit.wisc.edu> Message-ID: <696c56870804240657g53d24b68n2b4128f1b8480b8e@mail.gmail.com> Thanks Jesse. Your collection is great - but from a quick glance, it doesn't seem to match a double quote occurring only once. I am currently testing something like this: TCP|*|*|*|*|SMTP*|*|*|*"*"*"*|* $N$D30|Bad$ envelop$ from$ address TCP|*|*|*|*|SMTP*|*|*|*"*"*|* $Y TCP|*|*|*|*|SMTP*|*|*|*"*|* $N$D30|Bad$ envelop$ from$ address It's not perfect and can't get all unbalanced double quotes identified when the number of double quotes occurs more than 3 times. And the sort of regular expression that MS provides for mappings is quite brain dead. Can't figure out a way to precisely write a pattern that check for unbalanced quotes, without resorting to writing a C routine. Jeff On Thu, Apr 24, 2008 at 8:57 PM, Jesse Thompson wrote: > Hi Jeff, > > I'll let Eino address the gross crashing issue. But I have a workaround > for you. > > This FROM_ACCESS mapping will do the trick. After seeing a lot of garbage > in our queues due to 'mail from's with control characters, we did an > analysis (from our mail logs and searching around for what others are doing) > on a reasonable policy for valid characters in the local part of the mail > from. I know that we should technically be allowing more characters per the > RFC, but we've been running like this for a while and haven't had any > complaints. > > Jesse > > (in case you can't tell, the space character is covered by the range ascii > hex 0x01-0x21 (space is 0x20) which shows up visually as ^A-! note: the ^A > is what is printed in the terminal, but it's actually ascii hex 0x01. Let > me know if you need help typing that in (it's not ^ and A); it's possible to > do with vim and a bit of googling.) > > > FROM_ACCESS > ! > ! Require EHLO/HELO > *|*|*|*|*|SMTP/|*|*|*|* $N$D900|EHLO/HELO$ argument$ required > ! > ! Prohibit spaces in EHLO/HELO > *|*|*|*|*|SMTP/*$ *|*|*|*|* $NSpace$ not$ allowed$ in$ EHLO/HELO > ! > ! Prohibit invalid characters in anywhere in the uid > ! ascii hex: 01-21, 24, 25, 28, 29, 2c, 3a, 3b, 3c, 3e, 5b-5e, 60, 7b, 7d > *|*|*|*|*|*|*|*|*$[^A-!$%(),;^`\{\}]%*@*|* $Ninvalid$ character$ in$ mail$ > from > ! > ! Prohibit single character questionable characters > ! in the local part of the envelope from address > ! ascii hex: 23, 26, 27, 2a, 3b, 3f, 2b, 2d, 2e, 5f, 7e > *|*|*|*|*|*|*|*|$[#&'*=?+\-._\~]%@*|* $Ninvalid$ character$ in$ mail$ from > ! > ! Prohibit quoted single character questionable characters > ! in the local part of the envelope from address > ! ascii hex: 23, 26, 27, 2a, 3b, 3f, 2b, 2d, 2e, 5f, 7e > *|*|*|*|*|*|*|*|"$[#&'*=?+\-._\~]%"@*|* $Ninvalid$ character$ in$ mail$ > from > > > Jeff Chan wrote: > > > > > > > > > Hi, > > > > I started to experiment with Gross few days ago with SJSMS. I observed > > that Gross crashed regularly, without any error message, and thus left > > the server out of greylisting protection. > > > > I use Gross 1.0rc2 and found it crashses when a spammer sends > > something like this: > > > > MAIL FROM: SIZE=1000 > > > > The mail.log shows the MS takes spam"mer at dom> SIZE=1000 as the envelop > > address, without taking care of the unbalanced double quote. Although > > I think it's either a MS bug or at least a configuration issue, Gross > > shouldn't be crashing because of invalid inputs. > > > > Anyone encounters this issue? Or just me? And BTW, a little bit off > > topic, what's your experience dealing with invalid characters using > > the MS configuration files like mappings? > > > > Thanks. > > > > Jeff > > _______________________________________________ > > Gross mailing list > > Gross at lists.utu.fi > > https://lists.utu.fi/mailman/listinfo/gross > > > > -- > Jesse Thompson > Email/IM: jesse.thompson at doit.wisc.edu > From jesse.thompson at doit.wisc.edu Thu Apr 24 17:13:59 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Thu, 24 Apr 2008 09:13:59 -0500 Subject: [Gross] Space Character in Mail From Crashing Gross In-Reply-To: <696c56870804240657g53d24b68n2b4128f1b8480b8e@mail.gmail.com> References: <696c56870804240344x6bac796bmbf98230f2c88ef4f@mail.gmail.com> <481083D4.4030104@doit.wisc.edu> <696c56870804240657g53d24b68n2b4128f1b8480b8e@mail.gmail.com> Message-ID: <481095A7.9060907@doit.wisc.edu> Oh, sorry, I didn't read your message fully, and assumed you were talking about spaces, not quotes. It seems odd the gross would care about unbalanced quotes. Are you sure it's not crashing because of the space? Jesse Jeff Chan wrote: > Thanks Jesse. Your collection is great - but from a quick glance, it > doesn't seem to match a double quote occurring only once. > > I am currently testing something like this: > TCP|*|*|*|*|SMTP*|*|*|*"*"*"*|* $N$D30|Bad$ envelop$ from$ address > TCP|*|*|*|*|SMTP*|*|*|*"*"*|* $Y > TCP|*|*|*|*|SMTP*|*|*|*"*|* $N$D30|Bad$ envelop$ from$ address > > It's not perfect and can't get all unbalanced double quotes identified > when the number of double quotes occurs more than 3 times. And the > sort of regular expression that MS provides for mappings is quite > brain dead. Can't figure out a way to precisely write a pattern that > check for unbalanced quotes, without resorting to writing a C routine. > > Jeff > > On Thu, Apr 24, 2008 at 8:57 PM, Jesse Thompson > wrote: >> Hi Jeff, >> >> I'll let Eino address the gross crashing issue. But I have a workaround >> for you. >> >> This FROM_ACCESS mapping will do the trick. After seeing a lot of garbage >> in our queues due to 'mail from's with control characters, we did an >> analysis (from our mail logs and searching around for what others are doing) >> on a reasonable policy for valid characters in the local part of the mail >> from. I know that we should technically be allowing more characters per the >> RFC, but we've been running like this for a while and haven't had any >> complaints. >> >> Jesse >> >> (in case you can't tell, the space character is covered by the range ascii >> hex 0x01-0x21 (space is 0x20) which shows up visually as ^A-! note: the ^A >> is what is printed in the terminal, but it's actually ascii hex 0x01. Let >> me know if you need help typing that in (it's not ^ and A); it's possible to >> do with vim and a bit of googling.) >> >> >> FROM_ACCESS >> ! >> ! Require EHLO/HELO >> *|*|*|*|*|SMTP/|*|*|*|* $N$D900|EHLO/HELO$ argument$ required >> ! >> ! Prohibit spaces in EHLO/HELO >> *|*|*|*|*|SMTP/*$ *|*|*|*|* $NSpace$ not$ allowed$ in$ EHLO/HELO >> ! >> ! Prohibit invalid characters in anywhere in the uid >> ! ascii hex: 01-21, 24, 25, 28, 29, 2c, 3a, 3b, 3c, 3e, 5b-5e, 60, 7b, 7d >> *|*|*|*|*|*|*|*|*$[^A-!$%(),;^`\{\}]%*@*|* $Ninvalid$ character$ in$ mail$ >> from >> ! >> ! Prohibit single character questionable characters >> ! in the local part of the envelope from address >> ! ascii hex: 23, 26, 27, 2a, 3b, 3f, 2b, 2d, 2e, 5f, 7e >> *|*|*|*|*|*|*|*|$[#&'*=?+\-._\~]%@*|* $Ninvalid$ character$ in$ mail$ from >> ! >> ! Prohibit quoted single character questionable characters >> ! in the local part of the envelope from address >> ! ascii hex: 23, 26, 27, 2a, 3b, 3f, 2b, 2d, 2e, 5f, 7e >> *|*|*|*|*|*|*|*|"$[#&'*=?+\-._\~]%"@*|* $Ninvalid$ character$ in$ mail$ >> from >> >> >> Jeff Chan wrote: >> >>> >>> >>> Hi, >>> >>> I started to experiment with Gross few days ago with SJSMS. I observed >>> that Gross crashed regularly, without any error message, and thus left >>> the server out of greylisting protection. >>> >>> I use Gross 1.0rc2 and found it crashses when a spammer sends >>> something like this: >>> >>> MAIL FROM: SIZE=1000 >>> >>> The mail.log shows the MS takes spam"mer at dom> SIZE=1000 as the envelop >>> address, without taking care of the unbalanced double quote. Although >>> I think it's either a MS bug or at least a configuration issue, Gross >>> shouldn't be crashing because of invalid inputs. >>> >>> Anyone encounters this issue? Or just me? And BTW, a little bit off >>> topic, what's your experience dealing with invalid characters using >>> the MS configuration files like mappings? >>> >>> Thanks. >>> >>> Jeff >>> _______________________________________________ >>> Gross mailing list >>> Gross at lists.utu.fi >>> https://lists.utu.fi/mailman/listinfo/gross >>> >> -- >> Jesse Thompson >> Email/IM: jesse.thompson at doit.wisc.edu >> -- Jesse Thompson Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From jesse.thompson at doit.wisc.edu Thu Apr 24 17:18:11 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Thu, 24 Apr 2008 09:18:11 -0500 Subject: [Gross] Space Character in Mail From Crashing Gross In-Reply-To: <696c56870804240657g53d24b68n2b4128f1b8480b8e@mail.gmail.com> References: <696c56870804240344x6bac796bmbf98230f2c88ef4f@mail.gmail.com> <481083D4.4030104@doit.wisc.edu> <696c56870804240657g53d24b68n2b4128f1b8480b8e@mail.gmail.com> Message-ID: <481096A3.4090600@doit.wisc.edu> Jeff Chan wrote: > I am currently testing something like this: > TCP|*|*|*|*|SMTP*|*|*|*"*"*"*|* $N$D30|Bad$ envelop$ from$ address > TCP|*|*|*|*|SMTP*|*|*|*"*"*|* $Y > TCP|*|*|*|*|SMTP*|*|*|*"*|* $N$D30|Bad$ envelop$ from$ address > > It's not perfect and can't get all unbalanced double quotes identified > when the number of double quotes occurs more than 3 times. And the > sort of regular expression that MS provides for mappings is quite > brain dead. Can't figure out a way to precisely write a pattern that > check for unbalanced quotes, without resorting to writing a C routine. It's not regex; just simple string comparison. Wouldn't it be great if SJSMS supported regex? :-) Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From R.E.Sonneveld at sonnection.nl Thu Apr 24 18:36:12 2008 From: R.E.Sonneveld at sonnection.nl (Rolf E. Sonneveld) Date: Thu, 24 Apr 2008 17:36:12 +0200 Subject: [Gross] Space Character in Mail From Crashing Gross In-Reply-To: <481096A3.4090600@doit.wisc.edu> References: <696c56870804240344x6bac796bmbf98230f2c88ef4f@mail.gmail.com> <481083D4.4030104@doit.wisc.edu> <696c56870804240657g53d24b68n2b4128f1b8480b8e@mail.gmail.com> <481096A3.4090600@doit.wisc.edu> Message-ID: <4810A8EC.2050107@sonnection.nl> Jesse Thompson wrote: > Jeff Chan wrote: >> I am currently testing something like this: >> TCP|*|*|*|*|SMTP*|*|*|*"*"*"*|* $N$D30|Bad$ envelop$ from$ address >> TCP|*|*|*|*|SMTP*|*|*|*"*"*|* $Y >> TCP|*|*|*|*|SMTP*|*|*|*"*|* $N$D30|Bad$ envelop$ from$ address >> >> It's not perfect and can't get all unbalanced double quotes identified >> when the number of double quotes occurs more than 3 times. And the >> sort of regular expression that MS provides for mappings is quite >> brain dead. Can't figure out a way to precisely write a pattern that >> check for unbalanced quotes, without resorting to writing a C routine. > > It's not regex; just simple string comparison. Wouldn't it be great > if SJSMS supported regex? :-) Why don't we all file an RFE with Sun to support regex for the mappings file in the next version? /rolf From jesse.thompson at doit.wisc.edu Thu Apr 24 18:41:18 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Thu, 24 Apr 2008 10:41:18 -0500 Subject: [Gross] Space Character in Mail From Crashing Gross In-Reply-To: <4810A8EC.2050107@sonnection.nl> References: <696c56870804240344x6bac796bmbf98230f2c88ef4f@mail.gmail.com> <481083D4.4030104@doit.wisc.edu> <696c56870804240657g53d24b68n2b4128f1b8480b8e@mail.gmail.com> <481096A3.4090600@doit.wisc.edu> <4810A8EC.2050107@sonnection.nl> Message-ID: <4810AA1E.40505@doit.wisc.edu> Rolf E. Sonneveld wrote: > Jesse Thompson wrote: >> It's not regex; just simple string comparison. Wouldn't it be great >> if SJSMS supported regex? :-) > > Why don't we all file an RFE with Sun to support regex for the mappings > file in the next version? Good idea. Are you volunteering? :-) Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From jff.chan at gmail.com Thu Apr 24 23:09:43 2008 From: jff.chan at gmail.com (Jeff Chan) Date: Fri, 25 Apr 2008 04:09:43 +0800 Subject: [Gross] Space Character in Mail From Crashing Gross In-Reply-To: <481095A7.9060907@doit.wisc.edu> References: <696c56870804240344x6bac796bmbf98230f2c88ef4f@mail.gmail.com> <481083D4.4030104@doit.wisc.edu> <696c56870804240657g53d24b68n2b4128f1b8480b8e@mail.gmail.com> <481095A7.9060907@doit.wisc.edu> Message-ID: <696c56870804241309o4974c01fg3a0d9d9bd7f52605@mail.gmail.com> I think I found the bug with my rusty C programming skill. Try this: gclient sjsms '%20S' recipient 127.0.0.1 helo It will either cause strange "s=" entries to be recorded in the log on my Sun box or just crashes a Linux box. Seems that the %s that got passed in interfered with the normal format string processing with vnsprintf, that expects a further string value to get passed. Of course that didn't happen. The %20 is a hex encode of the space character. Spammer sends MAIL FROM: , MS thinks the envelop address is spam"mer at dom SIZE=XXX>, and passes spam"mer at dom%20SIZE=XXX> to grossd. Actually a simple adress like %s at dom is suffice to crash grossd, because the % character is not encoded by MS before sending to grossd for some reasons. Besides the envelop sender field, It also seems to affects the ricipient and helo field as well. It may also affect how tuples are entered into the bloom filter since grossd just takes what MS sends as is. Below are gdb stack trace that I think will help fixing the bug. For the meantime, I am working on script that auto restarts grossd when it dies. Program received signal SIGSEGV, Segmentation fault. [Switching to Thread -1290261584 (LWP 21965)] 0x00596eb8 in wcslen () from /lib/tls/libc.so.6 (gdb) where #0 0x00596eb8 in wcslen () from /lib/tls/libc.so.6 #1 0x00597f14 in wcsrtombs () from /lib/tls/libc.so.6 #2 0x005655ec in vfprintf () from /lib/tls/libc.so.6 #3 0x005829d6 in vsnprintf () from /lib/tls/libc.so.6 #4 0x0804b977 in logstr (level=6615028, fmt=0xb3181710 "a=greylist d=0 w=0 c=127.0.0.1 s=%20S r=r h=helo") at srvutils.c:50 #5 0x0804d416 in querylogwrite (q=0x81ae980) at worker.c:677 #6 0x0804d4c1 in finalize (status=0x81ae978) at worker.c:602 #7 0x08053baa in sjsms_connection (info=0x81adfd0, thread_ctx=0xb3182410, edict=0x81ae728) at worker_sjsms.c:270 #8 0x0805252e in thread_pool (arg=0x81ae590) at thread_pool.c:191 #9 0x00760371 in start_thread () from /lib/tls/libpthread.so.0 #10 0x005efffe in clone () from /lib/tls/libc.so.6 Jeff On 4/24/08, Jesse Thompson wrote: > Oh, sorry, I didn't read your message fully, and assumed you were > talking about spaces, not quotes. > > It seems odd the gross would care about unbalanced quotes. Are you sure > it's not crashing because of the space? > > Jesse > > > Jeff Chan wrote: > > Thanks Jesse. Your collection is great - but from a quick glance, it > > doesn't seem to match a double quote occurring only once. > > > > I am currently testing something like this: > > TCP|*|*|*|*|SMTP*|*|*|*"*"*"*|* $N$D30|Bad$ envelop$ from$ address > > TCP|*|*|*|*|SMTP*|*|*|*"*"*|* $Y > > TCP|*|*|*|*|SMTP*|*|*|*"*|* $N$D30|Bad$ envelop$ from$ address > > > > It's not perfect and can't get all unbalanced double quotes identified > > when the number of double quotes occurs more than 3 times. And the > > sort of regular expression that MS provides for mappings is quite > > brain dead. Can't figure out a way to precisely write a pattern that > > check for unbalanced quotes, without resorting to writing a C routine. > > > > Jeff > > > > On Thu, Apr 24, 2008 at 8:57 PM, Jesse Thompson > > wrote: > >> Hi Jeff, > >> > >> I'll let Eino address the gross crashing issue. But I have a workaround > >> for you. > >> > >> This FROM_ACCESS mapping will do the trick. After seeing a lot of > garbage > >> in our queues due to 'mail from's with control characters, we did an > >> analysis (from our mail logs and searching around for what others are > doing) > >> on a reasonable policy for valid characters in the local part of the mail > >> from. I know that we should technically be allowing more characters per > the > >> RFC, but we've been running like this for a while and haven't had any > >> complaints. > >> > >> Jesse > >> > >> (in case you can't tell, the space character is covered by the range > ascii > >> hex 0x01-0x21 (space is 0x20) which shows up visually as ^A-! note: the > ^A > >> is what is printed in the terminal, but it's actually ascii hex 0x01. > Let > >> me know if you need help typing that in (it's not ^ and A); it's possible > to > >> do with vim and a bit of googling.) > >> > >> > >> FROM_ACCESS > >> ! > >> ! Require EHLO/HELO > >> *|*|*|*|*|SMTP/|*|*|*|* $N$D900|EHLO/HELO$ argument$ required > >> ! > >> ! Prohibit spaces in EHLO/HELO > >> *|*|*|*|*|SMTP/*$ *|*|*|*|* $NSpace$ not$ allowed$ in$ EHLO/HELO > >> ! > >> ! Prohibit invalid characters in anywhere in the uid > >> ! ascii hex: 01-21, 24, 25, 28, 29, 2c, 3a, 3b, 3c, 3e, 5b-5e, 60, 7b, > 7d > >> *|*|*|*|*|*|*|*|*$[^A-!$%(),;^`\{\}]%*@*|* $Ninvalid$ character$ in$ > mail$ > >> from > >> ! > >> ! Prohibit single character questionable characters > >> ! in the local part of the envelope from address > >> ! ascii hex: 23, 26, 27, 2a, 3b, 3f, 2b, 2d, 2e, 5f, 7e > >> *|*|*|*|*|*|*|*|$[#&'*=?+\-._\~]%@*|* $Ninvalid$ character$ in$ mail$ > from > >> ! > >> ! Prohibit quoted single character questionable characters > >> ! in the local part of the envelope from address > >> ! ascii hex: 23, 26, 27, 2a, 3b, 3f, 2b, 2d, 2e, 5f, 7e > >> *|*|*|*|*|*|*|*|"$[#&'*=?+\-._\~]%"@*|* $Ninvalid$ character$ in$ mail$ > >> from > >> > >> > >> Jeff Chan wrote: > >> > >>> > >>> > >>> Hi, > >>> > >>> I started to experiment with Gross few days ago with SJSMS. I observed > >>> that Gross crashed regularly, without any error message, and thus left > >>> the server out of greylisting protection. > >>> > >>> I use Gross 1.0rc2 and found it crashses when a spammer sends > >>> something like this: > >>> > >>> MAIL FROM: SIZE=1000 > >>> > >>> The mail.log shows the MS takes spam"mer at dom> SIZE=1000 as the envelop > >>> address, without taking care of the unbalanced double quote. Although > >>> I think it's either a MS bug or at least a configuration issue, Gross > >>> shouldn't be crashing because of invalid inputs. > >>> > >>> Anyone encounters this issue? Or just me? And BTW, a little bit off > >>> topic, what's your experience dealing with invalid characters using > >>> the MS configuration files like mappings? > >>> > >>> Thanks. > >>> > >>> Jeff > >>> _______________________________________________ > >>> Gross mailing list > >>> Gross at lists.utu.fi > >>> https://lists.utu.fi/mailman/listinfo/gross > >>> > >> -- > >> Jesse Thompson > >> Email/IM: jesse.thompson at doit.wisc.edu > >> > > -- > Jesse Thompson > Email/IM: jesse.thompson at doit.wisc.edu > From antti at utu.fi Fri Apr 25 00:18:32 2008 From: antti at utu.fi (Antti Siira) Date: Fri, 25 Apr 2008 00:18:32 +0300 Subject: [Gross] Space Character in Mail From Crashing Gross In-Reply-To: <696c56870804241309o4974c01fg3a0d9d9bd7f52605@mail.gmail.com> References: <696c56870804240344x6bac796bmbf98230f2c88ef4f@mail.gmail.com> <481083D4.4030104@doit.wisc.edu> <696c56870804240657g53d24b68n2b4128f1b8480b8e@mail.gmail.com> <481095A7.9060907@doit.wisc.edu> <696c56870804241309o4974c01fg3a0d9d9bd7f52605@mail.gmail.com> Message-ID: <4810F928.1010209@utu.fi> Jeff Chan wrote: > Seems that the %s that got passed in interfered with the normal format > string processing with vnsprintf, that expects a further string value > to get passed. Of course that didn't happen. I committed a fix for this to the subversion repository and it would probably be a good idea to go through the other logging statements to make sure this bug is not found elsewhere. I'll look into it tomorrow, if time permits. -- Antti Siira From eino at utu.fi Mon Apr 28 08:54:05 2008 From: eino at utu.fi (Eino Tuominen) Date: Mon, 28 Apr 2008 08:54:05 +0300 Subject: [Gross] Space Character in Mail From Crashing Gross In-Reply-To: <696c56870804240344x6bac796bmbf98230f2c88ef4f@mail.gmail.com> References: <696c56870804240344x6bac796bmbf98230f2c88ef4f@mail.gmail.com> Message-ID: <4815667D.4070007@utu.fi> Jeff Chan wrote: > Hi, > > I started to experiment with Gross few days ago with SJSMS. I observed > that Gross crashed regularly, without any error message, and thus left > the server out of greylisting protection. Thank you for the report. The bug was a real brain fart in the new querylogging code, which Antti found and fixed. I went through the sources looking for the same kind of bugs and luckily this was the only one I found. Being away from computers for a couple of days seems to be the best debugging strategy. ;-) Believe me, this wasn't the only one... Anyway, I'll wrap up a yet another release candidate later today. -- Eino Tuominen From eino at utu.fi Mon Apr 28 14:06:11 2008 From: eino at utu.fi (Eino Tuominen) Date: Mon, 28 Apr 2008 14:06:11 +0300 Subject: [Gross] Last of the release candidates, I hope Message-ID: <4815AFA3.7040703@utu.fi> Hello, I was forced to release yet another release candidate, gross-1.0rc4. If it was just the logstr format bug I could have released 1.0 right away. Due to bad choice of a roff macro package for the man pages I have not yet received any feedback on man pages. Please, send me feedback (preferably through this list), so we can release the beast. Differences between rc3 and rc4: - rewritten man pages with perlpod. I tried to avoid dependency to pod2man, - fixed a format bug (issue #61) -- Eino Tuominen From jff.chan at gmail.com Wed Apr 30 14:49:14 2008 From: jff.chan at gmail.com (Jeff Chan) Date: Wed, 30 Apr 2008 19:49:14 +0800 Subject: [Gross] Patch to Greylist Bad Helo Message-ID: <696c56870804300449r19d63c29u56987e14856acae9@mail.gmail.com> Hi, For these few days, I went over the top and ran an experiement to greylist senders with bad helo strings using grossd. I did this because I found many obvious spams slipped through the RBLs undetected (as trusted). Those were the ones with bad helo strings, either with non-ascii characters, illegal domain name characters, non-fully qualified hostnames, or with IP addresses whether bracketed or not. I managed to patch grossd to check the validity of the helo string before checking at RBLs. So, it greylists senders even if they would be trusted. To illustrate the results, I examined one of my servers' log (it's a low volume server). About 58% sent valid helo, 42% didn't. Of those with valid helo, 12% were trusted, 11% were matched. Of those with invalid helo, only 0.0042% were matched. By this rate we could roughly estimate, with this helo check, we have increased the overall grossd hit performance by 5%. The code handled some cases with exception to RFCs. First, the code allows underscores, which is not uncommon for misconfigured hostnames. Second, even bracketed IP addresses that are RFC compliant, will get greylisted. I attach the diff against 1.0rc4 and hope that someone will find it useful. No warranty of course :-) P.S. The code short circuited the RBL checks, so I don't think it will honor any block_threshold, which I don't use BTW. But the advantage is it's faster and save processing power and bandwidth. That said, I think it should be relatively easy to overcome this. Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: gross-helo.diff Type: application/octet-stream Size: 1440 bytes Desc: not available URL: From eino at utu.fi Tue May 6 20:46:34 2008 From: eino at utu.fi (Eino Tuominen) Date: Tue, 06 May 2008 20:46:34 +0300 Subject: [Gross] Status update Message-ID: <4820997A.6010602@utu.fi> Hello, There is a double free() bug in worker_milter.c. See issue #62 for details and patch. Also, I'm bound to rewrite man pages once again with the painful good old man macro package. pod2man seems to be a bit too moving target to support properly. Besides, once the man pages are written they're gonna be quite easy to maintain regardless of the format. I'm still seeking feedback on man pages. It would be really nice if someone native English speaker found some precious time to review those. -- Eino Tuominen From jesse.thompson at doit.wisc.edu Tue May 13 18:39:48 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Tue, 13 May 2008 10:39:48 -0500 Subject: [Gross] Status update In-Reply-To: <4820997A.6010602@utu.fi> References: <4820997A.6010602@utu.fi> Message-ID: <4829B644.3000201@doit.wisc.edu> I made some minor changes to the man pages. They look great! Jesse Eino Tuominen wrote: > Hello, > > There is a double free() bug in worker_milter.c. See issue #62 for > details and patch. Also, I'm bound to rewrite man pages once again with > the painful good old man macro package. pod2man seems to be a bit too > moving target to support properly. Besides, once the man pages are > written they're gonna be quite easy to maintain regardless of the format. > > I'm still seeking feedback on man pages. It would be really nice if > someone native English speaker found some precious time to review those. > -- Jesse Thompson Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3353 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Tue May 13 18:55:07 2008 From: eino at utu.fi (Eino Tuominen) Date: Tue, 13 May 2008 18:55:07 +0300 Subject: [Gross] Status update In-Reply-To: <4829B644.3000201@doit.wisc.edu> References: <4820997A.6010602@utu.fi> <4829B644.3000201@doit.wisc.edu> Message-ID: <4829B9DB.2050904@utu.fi> Jesse Thompson wrote: > I made some minor changes to the man pages. They look great! Thanks alot! Now we are really really close... I know of three new installations, of which two runs Postfix. There's also two sites testing grossd with Sendmail and Milter, and a rather large site has already decided to put in in production (some 700000 messages a day). -- Eino Tuominen From eino at utu.fi Thu May 15 20:17:52 2008 From: eino at utu.fi (Eino Tuominen) Date: Thu, 15 May 2008 20:17:52 +0300 Subject: [Gross] Announce: Gross 1.0.0 released! Message-ID: <482C7040.4020907@utu.fi> Good evening (EET) everybody! I just uploaded 1.0.0 release to Google site. While testing it I found a drastic bug in rc5, so if you run rc5 you MUST upgrade (or downgrade to rc4.) I want to thank you all for your support, Gross would not be what it is today without your finding bugs and requesting changes. Gross has proven to be quite portable and very stable. In fact it has been extremely stable on Solaris, other platforms are getting there on a good pace. I know Grossd has been implemented on production systems running different MTA's and OS's, including Postfix, Sendmail, Exim and Sun Java Enterprise Messaging Server; Solaris, FreeBSD, NetBSD and Linux. Performance record so far has been over 100 million requests in just one day without any noticeable performance hit on the server. Way to go, Wisc.edu! Sendmail Milter support is in quite a good shape, too. There are at least two sites running Gross with Sendmail. New features: Version 1.0.0 * Prevent grossd from running as root (euid == 0). You may have to modify configuration and state file ownerships and/or permissions. Grossd tries to setuid() and setgid() to nobody if started by root. * New match shortcut logic. Now grossd will shortcut after a filter match only if no checks are configured (traditional greylister) or block_threshold == 0. * A lot of Milter related fixes. - changed filter_bits default back to 24, as stated in the docs. - added grey_reason configuration file option. See block_reason. Issues fixed: #48: Need man pages #59: grossd 0.9.3 leaks memory #60: Grossd should do all dns queries as fully qualified #61: Space Character in Mail From Crashing Gross #62: Double free() with milter #63: Apostrophe chokes old gcc #64: Garbage in logs with milter api -- Eino Tuominen From ew-lists at mailbox.com.pl Fri May 16 15:27:56 2008 From: ew-lists at mailbox.com.pl (Ernest Wypierowski) Date: Fri, 16 May 2008 14:27:56 +0200 Subject: [Gross] RFE: check = database, rereading configuration file Message-ID: <20080516122756.GF3560@aoe> Good morning, gross: Version 0.8.2 Maybe I misunderstanding the concept but if I am right the only way to do the classic's greylisting is to do --disable-dnsbl at the com- pilation time. If there is a way that I can achieve that in config file please give a shot. Why am I asking? I would like to change a policy during the night and be able to greylist all connections (beside dnswl). For that reason I am missing a con- figuration flag to modify the check from dnsbl to e.g. database. The next thing is that I've read the changelog (1.0.0) and didn't notice any sign of adopting e.g. HUP signal for rereading configura- tion file? Is there such a possibility? To sum up, is there a chance for doing: RFE1. changing the "check" flag in fly RFE2. HUP signal for rereading the config file Best regards, Ernest Wypierowski -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 2254 bytes Desc: not available URL: From eino at utu.fi Fri May 16 17:32:09 2008 From: eino at utu.fi (Eino Tuominen) Date: Fri, 16 May 2008 17:32:09 +0300 Subject: [Gross] RFE: check = database, rereading configuration file In-Reply-To: <20080516122756.GF3560@aoe> References: <20080516122756.GF3560@aoe> Message-ID: <482D9AE9.8040809@utu.fi> Ernest Wypierowski wrote: > Good morning, > > gross: Version 0.8.2 > > Maybe I misunderstanding the concept but if I am right the only way to > do the classic's greylisting is to do --disable-dnsbl at the com- > pilation time. > If there is a way that I can achieve that in config file please give a > shot. Just comment all checks out. I thought that setting grey_threshold = 0 would suffice, but it seems to be broken at the moment. If that worked you could combine traditional lister, but setting block_threshold would enable grossd to block too suspicious hosts. > I would like to change a policy during the night and be able to greylist > all connections (beside dnswl). For that reason I am missing a con- > figuration flag to modify the check from dnsbl to e.g. database. What MTA do you use? With SJSMS you could run Gross as a replicated pair and just restart grossd daemons one at a time. Just restart on host1 and check with telneting to status port that it's ok, and restart on host2. > The next thing is that I've read the changelog (1.0.0) and didn't > notice any sign of adopting e.g. HUP signal for rereading configura- > tion file? Is there such a possibility? I left this out from 1.0, because it would require some sort of locking or other means to alter the configuration on the fly. Now that all the world is no longer SJSMS this makes more sense. -- Eino Tuominen From ew-lists at mailbox.com.pl Fri May 16 17:55:53 2008 From: ew-lists at mailbox.com.pl (Ernest Wypierowski) Date: Fri, 16 May 2008 16:55:53 +0200 Subject: [Gross] RFE: check = database, rereading configuration file In-Reply-To: <482D9AE9.8040809@utu.fi> References: <20080516122756.GF3560@aoe> <482D9AE9.8040809@utu.fi> Message-ID: <20080516145553.GJ3560@aoe> On Fri, 16.05.2008 at 16:32:09, Eino Tuominen wrote: > Ernest Wypierowski wrote: > Just comment all checks out. I thought that setting grey_threshold = 0 > would suffice, but it seems to be broken at the moment. If that worked you > could combine traditional lister, but setting block_threshold would enable > grossd to block too suspicious hosts. If you comment out checks (Version 0.8.2) then all connections are "trust". >> I would like to change a policy during the night and be able to greylist >> all connections (beside dnswl). For that reason I am missing a con- >> figuration flag to modify the check from dnsbl to e.g. database. > > What MTA do you use? With SJSMS you could run Gross as a replicated pair > and just restart grossd daemons one at a time. Just restart on host1 and > check with telneting to status port that it's ok, and restart on host2. SJSMS. Gross, at the moment, is run in single mode. Best regards, Ernest Wypierowski From eino at utu.fi Fri May 16 18:12:16 2008 From: eino at utu.fi (Eino Tuominen) Date: Fri, 16 May 2008 18:12:16 +0300 Subject: [Gross] RFE: check = database, rereading configuration file In-Reply-To: <482D9AE9.8040809@utu.fi> References: <20080516122756.GF3560@aoe> <482D9AE9.8040809@utu.fi> Message-ID: <482DA450.3040100@utu.fi> Eino Tuominen wrote: > Ernest Wypierowski wrote: >> Good morning, >> >> gross: Version 0.8.2 >> >> Maybe I misunderstanding the concept but if I am right the only way to >> do the classic's greylisting is to do --disable-dnsbl at the com- >> pilation time. >> If there is a way that I can achieve that in config file please give a >> shot. > > Just comment all checks out. I thought that setting grey_threshold = 0 > would suffice, but it seems to be broken at the moment. If that worked > you could combine traditional lister, but setting block_threshold would > enable grossd to block too suspicious hosts. See Issue #67: http://code.google.com/p/gross/issues/detail?id=67 After applying the patch grossd will default to greylisting if grey_threshold = 0. That way you can still make checks and be able to block if necessary. -- Eino Tuominen From eino at utu.fi Fri May 16 19:04:44 2008 From: eino at utu.fi (Eino Tuominen) Date: Fri, 16 May 2008 19:04:44 +0300 Subject: [Gross] RFE: check = database, rereading configuration file In-Reply-To: <20080516145553.GJ3560@aoe> References: <20080516122756.GF3560@aoe> <482D9AE9.8040809@utu.fi> <20080516145553.GJ3560@aoe> Message-ID: <482DB09C.4040502@utu.fi> Ernest Wypierowski wrote: > On Fri, 16.05.2008 at 16:32:09, Eino Tuominen wrote: >> Ernest Wypierowski wrote: >> Just comment all checks out. I thought that setting grey_threshold = 0 >> would suffice, but it seems to be broken at the moment. If that worked you >> could combine traditional lister, but setting block_threshold would enable >> grossd to block too suspicious hosts. > > If you comment out checks (Version 0.8.2) then all connections are "trust". Oh. I suggest you try out the new 1.0.0 release, as 0.8.2 does not support that feature. But, you could do something like this with 0.8.2: configure and install two separate gross installations, /usr/local/gross-grey and /usr/local/gross-dnsbl Then, configure each to use same statefile. Of course, you have to be exremely careful to not allow them start while the other is running. -- Eino Tuominen From ew-lists at mailbox.com.pl Fri May 16 20:20:27 2008 From: ew-lists at mailbox.com.pl (Ernest Wypierowski) Date: Fri, 16 May 2008 19:20:27 +0200 Subject: [Gross] RFE: check = database, rereading configuration file In-Reply-To: <20080516145553.GJ3560@aoe> References: <20080516122756.GF3560@aoe> <482D9AE9.8040809@utu.fi> <20080516145553.GJ3560@aoe> Message-ID: <20080516172027.GE3376@aoe> On Fri, 16.05.2008 at 16:55:53, Ernest Wypierowski wrote: > On Fri, 16.05.2008 at 16:32:09, Eino Tuominen wrote: > > What MTA do you use? With SJSMS you could run Gross as a replicated pair > > and just restart grossd daemons one at a time. Just restart on host1 and > > check with telneting to status port that it's ok, and restart on host2. > > SJSMS. Gross, at the moment, is run in single mode. Another thing is that we will lose all historical connections after restart. So for some people/mta it could be greylisting for second time. Best regards, Ernest Wypierowski -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 2254 bytes Desc: not available URL: From ew-lists at mailbox.com.pl Fri May 16 20:21:15 2008 From: ew-lists at mailbox.com.pl (Ernest Wypierowski) Date: Fri, 16 May 2008 19:21:15 +0200 Subject: [Gross] RFE: check = database, rereading configuration file In-Reply-To: <482DB09C.4040502@utu.fi> References: <20080516122756.GF3560@aoe> <482D9AE9.8040809@utu.fi> <20080516145553.GJ3560@aoe> <482DB09C.4040502@utu.fi> Message-ID: <20080516172115.GF3376@aoe> On Fri, 16.05.2008 at 18:04:44, Eino Tuominen wrote: > Ernest Wypierowski wrote: >> On Fri, 16.05.2008 at 16:32:09, Eino Tuominen wrote: >>> Ernest Wypierowski wrote: >>> Just comment all checks out. I thought that setting grey_threshold = 0 >>> would suffice, but it seems to be broken at the moment. If that worked >>> you could combine traditional lister, but setting block_threshold would >>> enable grossd to block too suspicious hosts. >> If you comment out checks (Version 0.8.2) then all connections are >> "trust". > > Oh. I suggest you try out the new 1.0.0 release, as 0.8.2 does not support > that feature. And that is what I'm going to do. Best regards, Ernest Wypierowski From ernest.wypierowski at mailbox.com.pl Fri May 16 20:08:30 2008 From: ernest.wypierowski at mailbox.com.pl (Ernest Wypierowski) Date: Fri, 16 May 2008 19:08:30 +0200 Subject: [Gross] RFE: check = database, rereading configuration file In-Reply-To: <20080516145553.GJ3560@aoe> References: <20080516122756.GF3560@aoe> <482D9AE9.8040809@utu.fi> <20080516145553.GJ3560@aoe> Message-ID: <20080516170830.GC3376@aoe> On Fri, 16.05.2008 at 16:55:53, Ernest Wypierowski wrote: > On Fri, 16.05.2008 at 16:32:09, Eino Tuominen wrote: > > What MTA do you use? With SJSMS you could run Gross as a replicated pair > > and just restart grossd daemons one at a time. Just restart on host1 and > > check with telneting to status port that it's ok, and restart on host2. > > SJSMS. Gross, at the moment, is run in single mode. Another thing is that we will lose all historical connections after restart. So for some people/mta it could be greylisting for second time. Best regards, Ernest Wypierowski -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 2254 bytes Desc: not available URL: From ernest.wypierowski at mailbox.com.pl Fri May 16 20:03:17 2008 From: ernest.wypierowski at mailbox.com.pl (Ernest Wypierowski) Date: Fri, 16 May 2008 19:03:17 +0200 Subject: [Gross] RFE: check = database, rereading configuration file In-Reply-To: <482DB09C.4040502@utu.fi> References: <20080516122756.GF3560@aoe> <482D9AE9.8040809@utu.fi> <20080516145553.GJ3560@aoe> <482DB09C.4040502@utu.fi> Message-ID: <20080516170317.GA3376@aoe> On Fri, 16.05.2008 at 18:04:44, Eino Tuominen wrote: > Ernest Wypierowski wrote: >> On Fri, 16.05.2008 at 16:32:09, Eino Tuominen wrote: >>> Ernest Wypierowski wrote: >>> Just comment all checks out. I thought that setting grey_threshold = 0 >>> would suffice, but it seems to be broken at the moment. If that worked >>> you could combine traditional lister, but setting block_threshold would >>> enable grossd to block too suspicious hosts. >> If you comment out checks (Version 0.8.2) then all connections are >> "trust". > > Oh. I suggest you try out the new 1.0.0 release, as 0.8.2 does not support > that feature. And that is what I'm going to do. Best regards, Ernest Wypierowski -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 2254 bytes Desc: not available URL: From jesse.thompson at doit.wisc.edu Fri May 16 21:27:53 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Fri, 16 May 2008 13:27:53 -0500 Subject: [Gross] RFE: check = database, rereading configuration file In-Reply-To: <20080516122756.GF3560@aoe> References: <20080516122756.GF3560@aoe> Message-ID: <482DD229.1000105@doit.wisc.edu> Ernest Wypierowski wrote: > I would like to change a policy during the night and be able to greylist > all connections (beside dnswl). For that reason I am missing a con- > figuration flag to modify the check from dnsbl to e.g. database. Neat idea. Allowing for block_threshold and grey_threshold to be changed through external input, or given time-formatted values, would be an interesting feature request. Jesse -- Jesse Thompson Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3353 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Fri May 16 21:31:02 2008 From: eino at utu.fi (Eino Tuominen) Date: Fri, 16 May 2008 21:31:02 +0300 Subject: [Gross] RFE: check = database, rereading configuration file In-Reply-To: <20080516172027.GE3376@aoe> References: <20080516122756.GF3560@aoe> <482D9AE9.8040809@utu.fi> <20080516145553.GJ3560@aoe> <20080516172027.GE3376@aoe> Message-ID: <482DD2E6.8030806@utu.fi> Ernest Wypierowski wrote: > > Another thing is that we will lose all historical connections > after restart. So for some people/mta it could be greylisting for > second time. Configure a statefile, grossd will save the filters in the file. You will lose the update queue (grey_delay). If you configure synchronization you won't lose anything. In fact, you could configure a sync pair even in a single server situation. I haven't tested this, but running two separate grossd processes in a server like this should do it: server1.conf: sync_listen = 127.0.0.1 sync_peer = 127.0.0.2 server2.conf sync_listen = 127.0.0.2 sync_peer = 127.0.0.1 # ifconfig lo0 addif 127.0.0.2 up -- Eino Tuominen From pascal.maes at uclouvain.be Sat May 17 12:23:12 2008 From: pascal.maes at uclouvain.be (Pascal Maes) Date: Sat, 17 May 2008 11:23:12 +0200 Subject: [Gross] Question about gross and postfix Message-ID: <6467D8C0-C108-4616-99CE-1B329B9A05F7@uclouvain.be> hello, I have installed grossd to use it with postfix my main.cf file look like : # define the restriction classes # smtpd_restriction_classes = greylist_policy ... # define the greylist_policy # greylist_policy = check_policy_service inet:127.0.0.1:5525 # Recipient restrictions # smtpd_recipient_restrictions = reject_non_fqdn_recipient ... check_recipient_access hash:/etc/postfix/greylisting ... reject_rbl_client zen.dnsbl where /etc/postfix/greylisting contains the users or domains which allow the greylisting my.domain greylist_policy another.domain greylist_policy I can see that grossd is working as the logfile contains : May 17 11:07:14 smtp-1 grossd: #46e0a940: a=greylist d=0 w=2 c=118.136.19.250 s=dacuna at ugr.es r=dupont at dice.ucl.ac.be h=user-035a92fab2.firstmedia.com m=zen.dnsbl+2 but the message is not deferred as I see that it is rejected by the reject_rbl_client rule May 17 11:07:14 smtp-1 postfix/smtpd[9005]: NOQUEUE: reject: RCPT from unknown[118.136.19.250]: 554 5.7.1 Service unavailable; Client host [118.136.19.250] blocked using zen.dnsbl; http://www.spamhaus.org/query/bl?ip=118.136.19.250 ; from= to= proto=ESMTP helo= If I try the gclient, I have /usr/local/bin/gclient postfix dacuna at ugr.es dupont1 at dice.ucl.ac.be 118.136.19.250 user-035a92fab2.firstmedia.com action=defer_if_permit Please try again later. Where is the error(s) ? We are using postfix 2.5.1 Thanks -- Pascal From pascal.maes at uclouvain.be Sat May 17 12:29:54 2008 From: pascal.maes at uclouvain.be (Pascal Maes) Date: Sat, 17 May 2008 11:29:54 +0200 Subject: [Gross] Question about gross and postfix In-Reply-To: <6467D8C0-C108-4616-99CE-1B329B9A05F7@uclouvain.be> References: <6467D8C0-C108-4616-99CE-1B329B9A05F7@uclouvain.be> Message-ID: Le 17-mai-08 ? 11:23, Pascal Maes a ?crit : > hello, > > I have installed grossd to use it with postfix > my main.cf file look like : > > # define the restriction classes > # > smtpd_restriction_classes = > greylist_policy > ... > > # define the greylist_policy > # > greylist_policy = check_policy_service inet:127.0.0.1:5525 > > # Recipient restrictions > # > smtpd_recipient_restrictions = > > reject_non_fqdn_recipient > ... > check_recipient_access hash:/etc/postfix/greylisting > ... > reject_rbl_client zen.dnsbl > > where /etc/postfix/greylisting contains the users or domains which > allow the greylisting > > my.domain greylist_policy > another.domain greylist_policy > > I can see that grossd is working as the logfile contains : > May 17 11:07:14 smtp-1 grossd: #46e0a940: a=greylist d=0 w=2 > c=118.136.19.250 s=dacuna at ugr.es r=dupont at dice.ucl.ac.be > h=user-035a92fab2.firstmedia.com m=zen.dnsbl+2 > > but the message is not deferred as I see that it is rejected by the > reject_rbl_client rule > > May 17 11:07:14 smtp-1 postfix/smtpd[9005]: NOQUEUE: reject: RCPT from > unknown[118.136.19.250]: 554 5.7.1 Service unavailable; Client host > [118.136.19.250] blocked using zen.dnsbl; http://www.spamhaus.org/query/bl?ip=118.136.19.250 > ; from= to= proto=ESMTP > helo= > > If I try the gclient, I have > > /usr/local/bin/gclient postfix dacuna at ugr.es dupont1 at dice.ucl.ac.be > 118.136.19.250 user-035a92fab2.firstmedia.com > action=defer_if_permit Please try again later. > > Where is the error(s) ? > We are using postfix 2.5.1 > Well, I should read my postfix doc before asking that ! Is there a way to reject the message immediately with a code 4NN ? -- Pascal From pascal.maes at uclouvain.be Sat May 17 13:20:15 2008 From: pascal.maes at uclouvain.be (Pascal Maes) Date: Sat, 17 May 2008 12:20:15 +0200 Subject: [Gross] sync peer with multiple servers Message-ID: <9C5E5144-C32E-4F0D-AC51-181C0628450C@uclouvain.be> Hello, We have 4 servers (10.1.5.1, 10.1.5.2, 10.1.5.3, 10.1.5.4) Does the following configuration make sens ? for the machine 10.1.5.1, sync listen = 10.1.5.1 and sync peer = 10.1.5.4 for the machine 10.1.5.2, sync listen = 10.1.5.2 and sync peer = 10.1.5.1 for the machine 10.1.5.3, sync listen = 10.1.5.3 and sync peer = 10.1.5.2 for the machine 10.1.5.4, sync listen = 10.1.5.4 and sync peer = 10.1.5.3 Thanks, -- Pascal From eino at utu.fi Sat May 17 13:47:18 2008 From: eino at utu.fi (Eino Tuominen) Date: Sat, 17 May 2008 13:47:18 +0300 Subject: [Gross] Question about gross and postfix In-Reply-To: References: <6467D8C0-C108-4616-99CE-1B329B9A05F7@uclouvain.be> Message-ID: <482EB7B6.6000901@utu.fi> Pascal Maes wrote: > > Is there a way to reject the message immediately with a code 4NN ? Is this the same question as in comments on FutureIdeas wiki page? Right now grossd always returns "defer_if_permit %reason%". Is it enough to be able to configure "postfix_greylist_response" configuration option which would be "defer_if_permit %reason%" by default? Refer to analous sjsms options in grossd.conf manual. -- Eino Tuominen From eino at utu.fi Sat May 17 13:47:55 2008 From: eino at utu.fi (Eino Tuominen) Date: Sat, 17 May 2008 13:47:55 +0300 Subject: [Gross] sync peer with multiple servers In-Reply-To: <9C5E5144-C32E-4F0D-AC51-181C0628450C@uclouvain.be> References: <9C5E5144-C32E-4F0D-AC51-181C0628450C@uclouvain.be> Message-ID: <482EB7DB.7030006@utu.fi> Pascal Maes wrote: > Hello, > > We have 4 servers (10.1.5.1, 10.1.5.2, 10.1.5.3, 10.1.5.4) > > Does the following configuration make sens ? > > for the machine 10.1.5.1, sync listen = 10.1.5.1 and sync peer = > 10.1.5.4 > for the machine 10.1.5.2, sync listen = 10.1.5.2 and sync peer = > 10.1.5.1 > for the machine 10.1.5.3, sync listen = 10.1.5.3 and sync peer = > 10.1.5.2 > for the machine 10.1.5.4, sync listen = 10.1.5.4 and sync peer = > 10.1.5.3 Unfortunately, no. Grossd supports only two-way replication. But it shouldn't be a big problem as a single grossd is known to be able to handle over 100 million queries a day. So, configure and run two grossd instances and have your servers to query those. -- Eino Tuominen From eino at utu.fi Sat May 17 14:00:14 2008 From: eino at utu.fi (Eino Tuominen) Date: Sat, 17 May 2008 14:00:14 +0300 Subject: [Gross] RFE: check = database, rereading configuration file In-Reply-To: <482DD229.1000105@doit.wisc.edu> References: <20080516122756.GF3560@aoe> <482DD229.1000105@doit.wisc.edu> Message-ID: <482EBABE.4000500@utu.fi> Jesse Thompson wrote: > Ernest Wypierowski wrote: >> I would like to change a policy during the night and be able to greylist >> all connections (beside dnswl). For that reason I am missing a con- >> figuration flag to modify the check from dnsbl to e.g. database. > > Neat idea. Allowing for block_threshold and grey_threshold to be > changed through external input, or given time-formatted values, would be > an interesting feature request. More sofisticated methods can be implemented if configuration system gets a rewrite sometime in the future. But how about if checks could return bias to thresholds? A little like Puremessage rules can shift the final spam percentage by an absolute number. I mean that e.g. geoip check (not implemented yet, if you have good pointers to free databases, please share) could return bias if the mail comes in from some suspicious or highly trusted network. Some Finnish site that mostly exchanges mail with European countries could set rather strict thresholds and let geoip check raise the thresholds if sender ip is known to be European. Or time check could lower grey_threshold to 0 during night time. -- Eino Tuominen From pascal.maes at uclouvain.be Sat May 17 14:24:04 2008 From: pascal.maes at uclouvain.be (Pascal Maes) Date: Sat, 17 May 2008 13:24:04 +0200 Subject: [Gross] Question about gross and postfix In-Reply-To: <482EB7B6.6000901@utu.fi> References: <6467D8C0-C108-4616-99CE-1B329B9A05F7@uclouvain.be> <482EB7B6.6000901@utu.fi> Message-ID: <121D0E4E-41D8-4D56-8EC5-99C9473F486B@uclouvain.be> Le 17-mai-08 ? 12:47, Eino Tuominen a ?crit : > Pascal Maes wrote: >> Is there a way to reject the message immediately with a code 4NN ? > > Is this the same question as in comments on FutureIdeas wiki page? > Right now grossd always returns "defer_if_permit %reason%". Is it > enough to be able to configure "postfix_greylist_response" > configuration option which would be "defer_if_permit %reason%" by > default? Refer to analous sjsms options in grossd.conf manual. > > -- > Eino Tuominen Yes. If we could change the default to a "452 %reason%" it wil allow to not make the remaining checks which are time or resource comsuming. Keeping the %reason% is important to be able to analyze the logfile. Thanks -- Pascal From eino at utu.fi Sat May 17 14:35:18 2008 From: eino at utu.fi (Eino Tuominen) Date: Sat, 17 May 2008 14:35:18 +0300 Subject: [Gross] Question about gross and postfix In-Reply-To: <121D0E4E-41D8-4D56-8EC5-99C9473F486B@uclouvain.be> References: <6467D8C0-C108-4616-99CE-1B329B9A05F7@uclouvain.be> <482EB7B6.6000901@utu.fi> <121D0E4E-41D8-4D56-8EC5-99C9473F486B@uclouvain.be> Message-ID: <482EC2F6.5050602@utu.fi> Pascal Maes wrote: > > Yes. > If we could change the default to a "452 %reason%" it wil allow to not > make the remaining checks which are time or resource comsuming. > Keeping the %reason% is important to be able to analyze the logfile. I filed an RFE (issue #69) for this. -- Eino Tuominen From cite at incertum.net Sat May 17 23:17:24 2008 From: cite at incertum.net (Stefan =?utf-8?Q?F=C3=B6rster?=) Date: Sat, 17 May 2008 22:17:24 +0200 Subject: [Gross] sync peer with multiple servers Message-ID: <20080517201724.GA23468@mail.incertum.net> Eino Tuominen wrote: > Unfortunately, no. Grossd supports only two-way replication. But it > shouldn't be a big problem as a single grossd is known to be able to > handle over 100 million queries a day. > > So, configure and run two grossd instances and have your servers to > query those. Unfortunately, failure to contact a Postfix policy server will always result in a temporary failure (435, I believe). It is not possible to specify a "default" action. Ciao Stefan -- Stefan F?rster http://www.incertum.net/ Public Key: 0xBBE2A9E9 From eino at utu.fi Sun May 18 00:33:56 2008 From: eino at utu.fi (Eino Tuominen) Date: Sun, 18 May 2008 00:33:56 +0300 Subject: [Gross] sync peer with multiple servers In-Reply-To: <20080517201724.GA23468@mail.incertum.net> References: <20080517201724.GA23468@mail.incertum.net> Message-ID: <482F4F44.5010205@utu.fi> Stefan F?rster wrote: > > Unfortunately, failure to contact a Postfix policy server will always > result in a temporary failure (435, I believe). It is not possible to > specify a "default" action. That's the problem with Postfix, yes. The question was about a SJSMS environment, where failover is possible. BTW, Milter apparently can make use of round robin dns entries. How does Postfix behave if policy server name resolves to multiple ip addresses? -- Eino Tuominen From eino at utu.fi Mon May 19 07:27:49 2008 From: eino at utu.fi (Eino Tuominen) Date: Mon, 19 May 2008 07:27:49 +0300 Subject: [Gross] Opinion request Message-ID: <483101C5.60903@utu.fi> Hi, I'd like to get feedback about the default values of grey_reason and bock_reason. Now those two are "Please try again later" and "Bad reputation". I think the latter could be a bit more polite. So, suggestions, please! -- Eino Tuominen From jesse.thompson at doit.wisc.edu Tue May 20 19:19:08 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Tue, 20 May 2008 11:19:08 -0500 Subject: [Gross] RFE: check = database, rereading configuration file In-Reply-To: <482EBABE.4000500@utu.fi> References: <20080516122756.GF3560@aoe> <482DD229.1000105@doit.wisc.edu> <482EBABE.4000500@utu.fi> Message-ID: <4832F9FC.3080803@doit.wisc.edu> Eino Tuominen wrote: > Jesse Thompson wrote: >> Ernest Wypierowski wrote: >>> I would like to change a policy during the night and be able to greylist >>> all connections (beside dnswl). For that reason I am missing a con- >>> figuration flag to modify the check from dnsbl to e.g. database. >> >> Neat idea. Allowing for block_threshold and grey_threshold to be >> changed through external input, or given time-formatted values, would be >> an interesting feature request. > > More sofisticated methods can be implemented if configuration system > gets a rewrite sometime in the future. > > But how about if checks could return bias to thresholds? A little like > Puremessage rules can shift the final spam percentage by an absolute > number. It might be overkill. Puremessage deals with thousands of factors that bias the decision. Gross usually deals with less than 10. I'd rather see an ability to update some configuration variables without restarting grossd. This is more akin to how some of the SJSMS configutil variables can be updated without refreshing the server. This way you can add/remove dnsbls, change thresholds, modify error messages, etc, without disrupting the service. To accomplish Ernest's goal of lowering the grey_threshold during the night, one could put the command to change the value in the server's crontab. On the other hand, I don't think that I'd use this feature all that much, so I don't consider it a priority. > I mean that e.g. geoip check (not implemented yet, if you have good > pointers to free databases, please share) could return bias if the mail > comes in from some suspicious or highly trusted network. Some Finnish > site that mostly exchanges mail with European countries could set rather > strict thresholds and let geoip check raise the thresholds if sender ip > is known to be European. Or time check could lower grey_threshold to 0 > during night time. I think that geoip would be difficult to do live queries against, since the end-result will be dependent on your location in addition to the client_ip's location. Perhaps geoip would be better used to feed a locally hosted DNSBL or DNSWL that can then be queried by grossd. Jesse -- Jesse Thompson Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3353 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Tue May 20 19:27:45 2008 From: eino at utu.fi (Eino Tuominen) Date: Tue, 20 May 2008 19:27:45 +0300 Subject: [Gross] Announcement: Gross 1.0.1 available Message-ID: <4832FC01.50305@utu.fi> FYI, I just uploaded a patch release, it's available at the Google project site as usual. Version 1.0.1 * Two new configuration options: postfix_response_grey and postfix_response_block. Now it's possible to configure responses grossd sends in reply to Postfix policy delegation client. * Now grossd allows only the configured sync_peer to connect to sync server port. * When grey_threshold = 0 is set grossd defaults to greylisting. Now it's possible to combine traditional greylisting with blocking based on check results. Issues fixed: #65: Version number to log (RFE) #66: Compiler warnings on with 1.0.0 #67: Setting grey_threshold = 0 is not intuitive #68: grossd should accept sync connections only from the configured sync_peer #69: Implement postfix_response_grey and postfix_response_block (RFE) #70: Busyloop if no access to configured statefile -- Eino Tuominen From eino at utu.fi Tue May 20 21:30:43 2008 From: eino at utu.fi (Eino Tuominen) Date: Tue, 20 May 2008 21:30:43 +0300 Subject: [Gross] N-way cluster Message-ID: <483318D3.20606@utu.fi> Hi, I've received a lot of queries about supporting more that two node grossd clusters. Especially to Postfix admins this seems to be rather important issue. Anyway, I thought of telling you what we've been planning to do. Instead of replicating the database to more than two hosts, we plan to introduce a "satellite" or "slave" mode. There would still be a two node cluster that would take care of the bloom filters and replicate the database between those two. Master hosts could serve MTA's running checks normally or just take care of keeping the state information. A satellite grossd would do the tests normally, but instead of having the filters locally it would query from and send updates to the master hosts. Satellites would fetch most of the configuration (checks, weights etc.) from the masters, too. That would give you virtually unlimited scalability, and "n-node clusters". First we have to rewrite parts of the configuration code to support updating configuration on the fly. So, don't hold your breath... Feedback? -- Eino Tuominen From R.E.Sonneveld at sonnection.nl Thu May 22 00:33:30 2008 From: R.E.Sonneveld at sonnection.nl (Rolf E. Sonneveld) Date: Wed, 21 May 2008 23:33:30 +0200 Subject: [Gross] N-way cluster In-Reply-To: <483318D3.20606@utu.fi> References: <483318D3.20606@utu.fi> Message-ID: <4834952A.4030505@sonnection.nl> Eino Tuominen wrote: > Hi, > > I've received a lot of queries about supporting more that two node > grossd clusters. Especially to Postfix admins this seems to be rather > important issue. > > Anyway, I thought of telling you what we've been planning to do. Instead > of replicating the database to more than two hosts, we plan to introduce > a "satellite" or "slave" mode. There would still be a two node cluster > that would take care of the bloom filters and replicate the database > between those two. Master hosts could serve MTA's running checks > normally or just take care of keeping the state information. > > A satellite grossd would do the tests normally, but instead of having > the filters locally it would query from and send updates to the master > hosts. Satellites would fetch most of the configuration (checks, weights > etc.) from the masters, too. That would give you virtually unlimited > scalability, and "n-node clusters". > > First we have to rewrite parts of the configuration code to support > updating configuration on the fly. So, don't hold your breath... > > Feedback? > didn't see any replies. So here's my feedback: it sounds to me very promising! One comment: the satellites should have redundancy regarding the master host to use: e.g. of master host 1 is not reachable or doesn't respond fast enough, use master host 2. I'd suggest on a second track (in parallel) we try to make some PR for the gross. It is my experience that for products, that are widely used, it is not hard to justify the time investment needed to learn the product. With products that are not widely used it's often a matter of 'install, configure a minimum set, run and hope it will keep running'. Getting gross used by many people will more and more justify the time to learn the details of the product for all mail managers. For distributed environments I think it's a great product with many possibilities. I was thinking about a link on the Postfix HOWTO's page, for example, and probably one at the SJSMS wiki page, and I'm sure there are more locations to do some PR. Just my $0.02. Ehhh, ? 0.02 ;-) /rolf From eino at utu.fi Fri May 23 20:06:27 2008 From: eino at utu.fi (Eino Tuominen) Date: Fri, 23 May 2008 20:06:27 +0300 Subject: [Gross] Thread counts Message-ID: <4836F993.8040901@utu.fi> Hi, I'd like to get some stastics especially from the high volume sites. If you are allowed to, please send me the following info: From config file: filter_bits number_buffers check From ps: grossd thread count and memory consumption, and grossd statistics dump. Thanks! -- Eino Tuominen From natanael.copa at gmail.com Thu May 29 19:23:27 2008 From: natanael.copa at gmail.com (Natanael Copa) Date: Thu, 29 May 2008 18:23:27 +0200 Subject: [Gross] compile gross on uclibc Message-ID: <1212078207.4474.2.camel@nc.nor.wtbts.org> Hi, I managed to make gross-1.0.1 compile on linux/uclibc for embedded stuff. A tiny fix in configure.ac was needed since sem_init() needs -lpthread linker flag. Attatched is a trivial patch. please CC me as im not on the mailinglist. -nc -------------- next part -------------- A non-text attachment was scrubbed... Name: gross-1.0.1-pthread.patch Type: text/x-patch Size: 343 bytes Desc: not available URL: From natanael.copa at gmail.com Fri May 30 12:21:06 2008 From: natanael.copa at gmail.com (Natanael Copa) Date: Fri, 30 May 2008 11:21:06 +0200 Subject: [Gross] [PATCH] ./configure --enable-dnsbl will disable dnsbl Message-ID: <1212139266.4474.12.camel@nc.nor.wtbts.org> Hi, ./configure --enable-dnsbl will infact disable dnsbl. Attatched patch against current svn fixes that. thanks! -nc -------------- next part -------------- A non-text attachment was scrubbed... Name: gross-r487-enable-dnsbl.patch Type: text/x-patch Size: 452 bytes Desc: not available URL: From eino at utu.fi Fri May 30 12:49:20 2008 From: eino at utu.fi (Eino Tuominen) Date: Fri, 30 May 2008 12:49:20 +0300 Subject: [Gross] [PATCH] ./configure --enable-dnsbl will disable dnsbl In-Reply-To: <1212139266.4474.12.camel@nc.nor.wtbts.org> References: <1212139266.4474.12.camel@nc.nor.wtbts.org> Message-ID: <483FCDA0.8000708@utu.fi> Natanael Copa wrote: > Hi, > > ./configure --enable-dnsbl will infact disable dnsbl. > > Attatched patch against current svn fixes that. > > thanks! Thanks for the patch, I'll include it in 1.0.2. (and check if --disable-milter has the same problem) -- Eino Tuominen From eino at utu.fi Fri May 30 14:24:04 2008 From: eino at utu.fi (Eino Tuominen) Date: Fri, 30 May 2008 14:24:04 +0300 Subject: [Gross] Problems with FreeBSD port definition Message-ID: <483FE3D4.4000203@utu.fi> Hi, There was a problem with the latest FreeBSD port definition package. A fault in configure lead to compiling grossd always with --disable-dnsbl and --enable-milter regardless of the config options. I uploaded a new port definition package to work around the problem. The real problem will be fixed in next grossd release, this is just a workaround to not use the broken --enable-dnsbl and --disable-milter configure options. -- Eino Tuominen From natanael.copa at gmail.com Fri May 30 15:27:41 2008 From: natanael.copa at gmail.com (Natanael Copa) Date: Fri, 30 May 2008 14:27:41 +0200 Subject: [Gross] [PATCH] ./configure --enable-dnsbl will disable dnsbl In-Reply-To: <483FCDA0.8000708@utu.fi> References: <1212139266.4474.12.camel@nc.nor.wtbts.org> <483FCDA0.8000708@utu.fi> Message-ID: <1212150461.4474.19.camel@nc.nor.wtbts.org> On Fri, 2008-05-30 at 12:49 +0300, Eino Tuominen wrote: > Natanael Copa wrote: > > Hi, > > > > ./configure --enable-dnsbl will infact disable dnsbl. > > > > Attatched patch against current svn fixes that. > > > > thanks! > > Thanks for the patch, I'll include it in 1.0.2. (and check if > --disable-milter has the same problem) > Could you please also apply the previous patch I posted, which checks if sem_init() needs -lpthread linker flag. It's needed for uclibc. thanks! -nc From eino at utu.fi Fri May 30 18:13:37 2008 From: eino at utu.fi (Eino Tuominen) Date: Fri, 30 May 2008 18:13:37 +0300 Subject: [Gross] [PATCH] ./configure --enable-dnsbl will disable dnsbl In-Reply-To: <1212150461.4474.19.camel@nc.nor.wtbts.org> References: <1212139266.4474.12.camel@nc.nor.wtbts.org> <483FCDA0.8000708@utu.fi> <1212150461.4474.19.camel@nc.nor.wtbts.org> Message-ID: <484019A1.8080000@utu.fi> Natanael Copa wrote: > > Could you please also apply the previous patch I posted, which checks if > sem_init() needs -lpthread linker flag. It's needed for uclibc. Sure. I just forgot to respond to your mail, sorry about that. -- Eino Tuominen From eino at utu.fi Fri May 30 20:05:13 2008 From: eino at utu.fi (Eino Tuominen) Date: Fri, 30 May 2008 20:05:13 +0300 Subject: [Gross] [PATCH] ./configure --enable-dnsbl will disable dnsbl In-Reply-To: <1212150461.4474.19.camel@nc.nor.wtbts.org> References: <1212139266.4474.12.camel@nc.nor.wtbts.org> <483FCDA0.8000708@utu.fi> <1212150461.4474.19.camel@nc.nor.wtbts.org> Message-ID: <484033C9.9070803@utu.fi> Natanael Copa wrote: > On Fri, 2008-05-30 at 12:49 +0300, Eino Tuominen wrote: >> Natanael Copa wrote: >>> Hi, >>> >>> ./configure --enable-dnsbl will infact disable dnsbl. >>> >>> Attatched patch against current svn fixes that. >>> >>> thanks! >> Thanks for the patch, I'll include it in 1.0.2. (and check if >> --disable-milter has the same problem) >> > > Could you please also apply the previous patch I posted, which checks if > sem_init() needs -lpthread linker flag. It's needed for uclibc. I just committed these. --disable-dnsbl needed a bit more work, but now it (and --enable-milter) seems to work correctly. -- Eino Tuominen From jesse.thompson at doit.wisc.edu Tue Jun 10 16:48:36 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Tue, 10 Jun 2008 08:48:36 -0500 Subject: [Gross] hermes.sun.com Message-ID: <484E8634.6030801@doit.wisc.edu> Anyone else seeing the Sun newsletters (coming from hermes.sun.com) getting greylisted by gross? It seems that the IP's ASN is on dnsbl-3.uceprotect.net (which means that the ASN has more than 100 IPs listed on dnsbl-1.uceprotect.net). Also, Sun isn't actually attempting to retry the messages. When I asked them about it, they said that they didn't know which of our servers to retry to (they can't correctly interpret MX records.) It's kinda annoying that Sun is being so dense. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3353 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Tue Jun 10 19:19:21 2008 From: eino at utu.fi (Eino Tuominen) Date: Tue, 10 Jun 2008 19:19:21 +0300 Subject: [Gross] hermes.sun.com In-Reply-To: <484E8634.6030801@doit.wisc.edu> References: <484E8634.6030801@doit.wisc.edu> Message-ID: <484EA989.10506@utu.fi> Jesse Thompson wrote: > > Also, Sun isn't actually attempting to retry the messages. When I asked > them about it, they said that they didn't know which of our servers to > retry to (they can't correctly interpret MX records.) Weird response from Sun, indeed. How did they know which server to try in the first place? > It's kinda annoying that Sun is being so dense. If you still have the response, please forward it to me. I met an email administrator from Sun at the comms summit in Brussels last week, I could ask him if he has some technical details on the matter. It was funny to see how widely spread Gross actually is. Either the Google's download counters are skewed or svn is used more than I've thought. E.g. couple of major Polish ISP's have implemented it and there were sites handling tens of millions of messages each day. It gave me a lot of motivation to work on developing Gross. From jesse.thompson at doit.wisc.edu Tue Jun 10 19:34:24 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Tue, 10 Jun 2008 11:34:24 -0500 Subject: [Gross] hermes.sun.com In-Reply-To: <484EA989.10506@utu.fi> References: <484E8634.6030801@doit.wisc.edu> <484EA989.10506@utu.fi> Message-ID: <484EAD10.10408@doit.wisc.edu> Eino Tuominen wrote: > Jesse Thompson wrote: >> >> Also, Sun isn't actually attempting to retry the messages. When I asked >> them about it, they said that they didn't know which of our servers to >> retry to (they can't correctly interpret MX records.) > > Weird response from Sun, indeed. How did they know which server to try > in the first place? Just fired off an email to info-ims. Maybe one of the Sun guys on the list will clue them in. >> It's kinda annoying that Sun is being so dense. > > If you still have the response, please forward it to me. I met an email > administrator from Sun at the comms summit in Brussels last week, I > could ask him if he has some technical details on the matter. It was > funny to see how widely spread Gross actually is. Either the Google's > download counters are skewed or svn is used more than I've thought. E.g. > couple of major Polish ISP's have implemented it and there were sites > handling tens of millions of messages each day. It gave me a lot of > motivation to work on developing Gross. Sent. They're currently asking me to whitelist their IPs from greylisting. Apparently, they have to do this with a lot of customers. I don't want to do it. Thanks, Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3353 bytes Desc: S/MIME Cryptographic Signature URL: From hege at hege.li Tue Jun 10 20:14:43 2008 From: hege at hege.li (Henrik Krohns) Date: Tue, 10 Jun 2008 20:14:43 +0300 Subject: [Gross] hermes.sun.com In-Reply-To: <484EAD10.10408@doit.wisc.edu> References: <484E8634.6030801@doit.wisc.edu> <484EA989.10506@utu.fi> <484EAD10.10408@doit.wisc.edu> Message-ID: <20080610171442.GA27123@posti.hege.li> On Tue, Jun 10, 2008 at 11:34:24AM -0500, Jesse Thompson wrote: > > They're currently asking me to whitelist their IPs from greylisting. > Apparently, they have to do this with a lot of customers. I don't want to > do it. Why would you want to greylist legimate senders? You are just wasting everybodys resources. You should simply whitelist all confirmed reverse-addresses matching .sun.com or atleast use dnswl.org data (hermes is included there also). From jesse.thompson at doit.wisc.edu Tue Jun 10 21:11:54 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Tue, 10 Jun 2008 13:11:54 -0500 Subject: [Gross] hermes.sun.com In-Reply-To: <20080610171442.GA27123@posti.hege.li> References: <484E8634.6030801@doit.wisc.edu> <484EA989.10506@utu.fi> <484EAD10.10408@doit.wisc.edu> <20080610171442.GA27123@posti.hege.li> Message-ID: <484EC3EA.3030504@doit.wisc.edu> Henrik Krohns wrote: > On Tue, Jun 10, 2008 at 11:34:24AM -0500, Jesse Thompson wrote: >> They're currently asking me to whitelist their IPs from greylisting. >> Apparently, they have to do this with a lot of customers. I don't want to >> do it. > > Why would you want to greylist legimate senders? You are just wasting > everybodys resources. I don't want to greylist legitimate senders. Ideally, Sun should get themselves off of blacklists if they expect mail to flow efficiently to everyone (we're very conservative compared to others.) Their narcissistic process of making customers manually whitelist their servers is wasting more resources than whatever resources would be spent by Sun retrying on 4xx. > You should simply whitelist all confirmed reverse-addresses matching > .sun.com or atleast use dnswl.org data (hermes is included there also). Whitelisting all confirmed reverse-addresses matching .sun.com is still too manual. But, it is better than whitelisting their individual IP addresses. DNSWL is a good idea too, which I've been planning to implement in gross for some time. But I still would like Sun to fix their issues. Jesse -- Jesse Thompson Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3353 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Wed Jun 11 10:07:11 2008 From: eino at utu.fi (Eino Tuominen) Date: Wed, 11 Jun 2008 10:07:11 +0300 Subject: [Gross] hermes.sun.com In-Reply-To: <484EC3EA.3030504@doit.wisc.edu> References: <484E8634.6030801@doit.wisc.edu> <484EA989.10506@utu.fi> <484EAD10.10408@doit.wisc.edu> <20080610171442.GA27123@posti.hege.li> <484EC3EA.3030504@doit.wisc.edu> Message-ID: <484F799F.9060303@utu.fi> Jesse Thompson wrote: > > I don't want to greylist legitimate senders. Ideally, Sun should get > themselves off of blacklists if they expect mail to flow efficiently to > everyone (we're very conservative compared to others.) Their > narcissistic process of making customers manually whitelist their > servers is wasting more resources than whatever resources would be spent > by Sun retrying on 4xx. Not to mention their implementation of SMTP sending out these newsletters is grossly (sic) violating the RFC: 4.5.4.1 Sending Strategy ... while mail that cannot be transmitted immediately MUST be queued and periodically retried by the sender. ... -- Eino Tuominen From Arnis.Juraga at biti.lv Fri Jun 27 09:57:51 2008 From: Arnis.Juraga at biti.lv (Arnis Juraga) Date: Fri, 27 Jun 2008 09:57:51 +0300 Subject: [Gross] High CPU load on small system. Message-ID: <48648F6F.1070801@biti.lv> Hello! I have installed Gross on test server, using sendmail, clamav and spamassassin, pop3 and imap. Gross is working fine, but I have permanent CPU load ~60-80 %by grossd. This is not an high performance server (Pentium 4, 1.7GHz), mail statistics on that server is about 10 messages PER MINUTE. I have 1GB ram, (which is fully used, so). Grosss -d shows no errors, i have limited pool_maxthreads =10 number_buffers = 4 but nothing changed. What can i do to trace or resolve problem? Arnis From eino at utu.fi Fri Jun 27 10:11:15 2008 From: eino at utu.fi (Eino Tuominen) Date: Fri, 27 Jun 2008 10:11:15 +0300 Subject: [Gross] High CPU load on small system. In-Reply-To: <48648F6F.1070801@biti.lv> References: <48648F6F.1070801@biti.lv> Message-ID: <48649293.3070106@utu.fi> Arnis Juraga wrote: > Hello! > > I have installed Gross on test server, using sendmail, clamav and > spamassassin, pop3 and imap. Gross is working fine, but I have permanent > CPU load ~60-80 %by grossd. > This is not an high performance server (Pentium 4, 1.7GHz), mail > statistics on that server is about 10 messages PER MINUTE. > > I have 1GB ram, (which is fully used, so). Grosss -d shows no errors, i > have limited > pool_maxthreads =10 > number_buffers = 4 > > but nothing changed. > > What can i do to trace or resolve problem? Hmm, that sounds really weird. What OS are you running on? If you have truss or ktrace or something similar you could see if grossd is doing any system calls. Do you have dnsbl configured? If you do, does the behaviour change if you compile gross with --disable-dnsbl? -- Eino Tuominen From Arnis.Juraga at biti.lv Tue Jul 1 17:29:30 2008 From: Arnis.Juraga at biti.lv (Arnis Juraga) Date: Tue, 01 Jul 2008 17:29:30 +0300 Subject: [Gross] High CPU load on small system. In-Reply-To: <48649293.3070106@utu.fi> References: <48648F6F.1070801@biti.lv> <48649293.3070106@utu.fi> Message-ID: <486A3F4A.7010000@biti.lv> Hi! test server has Slackware 10.2 It has not utilitys You adviced, i used strace. (11292 - is PID for "mad" grossd child process (eating 98% CPU)) Grants# strace -p 11292 nanosleep({1074233051, 1075662836}, 0xbf7ffab8) = -1 EINVAL (Invalid argument) nanosleep({1074233051, 1075662836}, 0xbf7ffab8) = -1 EINVAL (Invalid argument) nanosleep({1074233051, 1075662836}, 0xbf7ffab8) = -1 EINVAL (Invalid argument) nanosleep({1074233051, 1075662836}, 0xbf7ffab8) = -1 EINVAL (Invalid argument) nanosleep({1074233051, 1075662836}, 0xbf7ffab8) = -1 EINVAL (Invalid argument) nanosleep({1074233051, 1075662836}, 0xbf7ffab8) = -1 EINVAL (Invalid argument) nanosleep({1074233051, 1075662836}, 0xbf7ffab8) = -1 EINVAL (Invalid argument) nanosleep({1074233051, 1075662836}, 0xbf7ffab8) = -1 EINVAL (Invalid argument) nanosleep({1074233051, 1075662836}, 0xbf7ffab8) = -1 EINVAL (Invalid argument) ...... Can it help? Arnis Juraga Eino Tuominen wrote: > Arnis Juraga wrote: >> Hello! >> >> I have installed Gross on test server, using sendmail, clamav and >> spamassassin, pop3 and imap. Gross is working fine, but I have >> permanent CPU load ~60-80 %by grossd. >> This is not an high performance server (Pentium 4, 1.7GHz), mail >> statistics on that server is about 10 messages PER MINUTE. >> >> I have 1GB ram, (which is fully used, so). Grosss -d shows no errors, >> i have limited >> pool_maxthreads =10 >> number_buffers = 4 >> >> but nothing changed. >> >> What can i do to trace or resolve problem? > > Hmm, that sounds really weird. What OS are you running on? If you have > truss or ktrace or something similar you could see if grossd is doing > any system calls. Do you have dnsbl configured? If you do, does the > behaviour change if you compile gross with --disable-dnsbl? > From eino at utu.fi Sun Jul 27 19:39:18 2008 From: eino at utu.fi (Eino Tuominen) Date: Sun, 27 Jul 2008 19:39:18 +0300 Subject: [Gross] Gross at USENIX LISA 2008 Message-ID: <488CA4B6.9050609@utu.fi> Hi all! I've almost through my summer holidays (still a few days left), and during the rainy days I applied for a poster session at USENIX LISA 2008. I just got the email saying my poster has been accepted! So, see you at LISA in November. -- Eino Tuominen From jesse.thompson at doit.wisc.edu Mon Jul 28 19:41:05 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Mon, 28 Jul 2008 11:41:05 -0500 Subject: [Gross] Gross at USENIX LISA 2008 In-Reply-To: <488CA4B6.9050609@utu.fi> References: <488CA4B6.9050609@utu.fi> Message-ID: <488DF6A1.80205@doit.wisc.edu> Congrats! I don't think I'll be there though. :-( Jesse Eino Tuominen wrote: > Hi all! > > I've almost through my summer holidays (still a few days left), and > during the rainy days I applied for a poster session at USENIX LISA > 2008. I just got the email saying my poster has been accepted! > > So, see you at LISA in November. > -- Jesse Thompson Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3353 bytes Desc: S/MIME Cryptographic Signature URL: From steeeeeveee at gmx.net Sat Aug 2 19:45:15 2008 From: steeeeeveee at gmx.net (Steve) Date: Sat, 02 Aug 2008 18:45:15 +0200 Subject: [Gross] gross 1.0.1 milter and Postfix Message-ID: <20080802164515.149420@gmx.net> Hello list I am using gross 1.0.1 as an Milter in Postfix 2.5.3. It looks that all is working as expected (okay... I once and just once had an segfault with it, but that's all so far). Now I have a question: How does gross 1.0.1 Milter knows that a mail is locally originating? I don't see any option in gross to disable it on mails originating from local addresses and/or I don't see any option in gross to disable it for authenticated users. With authenticated I mean users that succeeded SMTP AUTH ({auth_authen} Milter macro) and/or users that use STARTTLS with a verified client certificate ({verify} and {cert_subject} macro). Is there a way to bypass gross for those conditions? Does any one know that? What about white listing certain client IP's and/or recipient and/or senders? Is there a possibility to do that in gross (beside adding them in a local dnswl)? btw: I know, I know. I should/could use the Postfix policy with gross. But I just wanted to see if the Milter would work with Postfix. And it does :) But probably I will switch gross to use Postfix policy possibility. // SteveB -- Der GMX SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen! Ideal f?r Modem und ISDN: http://www.gmx.net/de/go/smartsurfer From eino at utu.fi Sat Aug 2 19:59:06 2008 From: eino at utu.fi (Eino Tuominen) Date: Sat, 02 Aug 2008 19:59:06 +0300 Subject: [Gross] gross 1.0.1 milter and Postfix In-Reply-To: <20080802164515.149420@gmx.net> References: <20080802164515.149420@gmx.net> Message-ID: <4894925A.2050101@utu.fi> Steve wrote: > Hello list > > I am using gross 1.0.1 as an Milter in Postfix 2.5.3. It looks that > all is working as expected (okay... I once and just once had an > segfault with it, but that's all so far). Well, even the one segfault is too much.. ;-) On what platform do you run your email stack? > Now I have a question: How does gross 1.0.1 Milter knows that a mail > is locally originating? I don't see any option in gross to disable it > on mails originating from local addresses and/or I don't see any > option in gross to disable it for authenticated users. With > authenticated I mean users that succeeded SMTP AUTH ({auth_authen} > Milter macro) and/or users that use STARTTLS with a verified client > certificate ({verify} and {cert_subject} macro). Is there a way to > bypass gross for those conditions? Does any one know that? I'm afraid that funcionality is not (yet) implemented in grossd. It should be pretty straight forward to iplement those things in worker_milter.c. > What about white listing certain client IP's and/or recipient and/or > senders? Is there a possibility to do that in gross (beside adding > them in a local dnswl)? A local dnswl is the only option, currently. Senders could be whitelisted with a local rhsbl with a negative weight. Now, I'm uncertain if I have ever tested negative weights thorougly, but It-Should-Work(TM). > btw: I know, I know. I should/could use the Postfix policy with > gross. But I just wanted to see if the Milter would work with > Postfix. And it does :) But probably I will switch gross to use > Postfix policy possibility. I'm going to investigate milter possibilities more after I've got our JES to speak milter with gross. One interesting possibility would be to extract URLs from message bodies and test them against Spamhaus SBL. -- Eino Tuominen From steeeeeveee at gmx.net Sat Aug 2 19:59:22 2008 From: steeeeeveee at gmx.net (Steve) Date: Sat, 02 Aug 2008 18:59:22 +0200 Subject: [Gross] gross 1.0.1 dnsbl/dnswl Message-ID: <20080802165922.149420@gmx.net> Hello list Is there an possibility in gross to give different weight for a dnsbl/dnswl response? I mean: There are dnsbl's/dnswl's out there giving different results depending on the query you do. For example hostkarma.junkemailfilter.com sends following responses back: 127.0.0.1 - whilelist - trusted nonspam 127.0.0.2 - blacklist - block spam 127.0.0.3 - yellowlist - mix of spam and nonspam 127.0.0.4 - brownlist - all spam - but not yet enough to blacklist Another example would be lookups to zz.countries.nerd.dk (http://countries.nerd.dk/more.html). They return the ISO 3166 A 2 country code of the IP. This is a great way to just make one single lookup and either white list all countries you don't want to greylist or to give with just one single lookup a weight to all the countries you would like to block/greylist. Can gross handle such responses or does it just looks if the lookup gives an response or not? What about the different weights depending on the response? Is that possible with gross? // SteveB -- GMX Kostenlose Spiele: Einfach online spielen und Spa? haben mit Pastry Passion! http://games.entertainment.gmx.net/de/entertainment/games/free/puzzle/6169196 From eino at utu.fi Sat Aug 2 20:15:57 2008 From: eino at utu.fi (Eino Tuominen) Date: Sat, 02 Aug 2008 20:15:57 +0300 Subject: [Gross] gross 1.0.1 dnsbl/dnswl In-Reply-To: <20080802165922.149420@gmx.net> References: <20080802165922.149420@gmx.net> Message-ID: <4894964D.3060102@utu.fi> Steve wrote: > Hello list > > Is there an possibility in gross to give different weight for a > dnsbl/dnswl response? I mean: There are dnsbl's/dnswl's out there > giving different results depending on the query you do. For example > hostkarma.junkemailfilter.com sends following responses back: No, there is no way to do that. It would be an excellent enhancement, though. Right now I can't come up with a clean way to express that in config file. Maybe something like dnsbl=foo.org;5;127.0.0.1 dnsbl=foo.org;3;127.0.0.2 dnswl=foo.org;127.0.0.3 The config system really needs a rewrite, the syntax is getting quite patchy... > Another example would be lookups to zz.countries.nerd.dk > (http://countries.nerd.dk/more.html). They return the ISO 3166 A 2 > country code of the IP. This is a great way to just make one single > lookup and either white list all countries you don't want to greylist > or to give with just one single lookup a weight to all the countries > you would like to block/greylist. A good idea, too. -- Eino Tuominen From steeeeeveee at gmx.net Sat Aug 2 20:32:42 2008 From: steeeeeveee at gmx.net (Steve) Date: Sat, 02 Aug 2008 19:32:42 +0200 Subject: [Gross] gross 1.0.1 milter and Postfix In-Reply-To: <4894925A.2050101@utu.fi> References: <20080802164515.149420@gmx.net> <4894925A.2050101@utu.fi> Message-ID: <20080802173242.149400@gmx.net> -------- Original-Nachricht -------- > Datum: Sat, 02 Aug 2008 19:59:06 +0300 > Von: Eino Tuominen > An: Steve > CC: gross at lists.utu.fi > Betreff: Re: [Gross] gross 1.0.1 milter and Postfix > Steve wrote: > > Hello list > > > > I am using gross 1.0.1 as an Milter in Postfix 2.5.3. It looks that > > all is working as expected (okay... I once and just once had an > > segfault with it, but that's all so far). > > Well, even the one segfault is too much.. ;-) > Well... that's life. What should I say? Here the output while running: mail ~ # /usr/sbin/grossd -d Sat Aug 2 14:43:13 2008 #b7dea6c0: grossd version 1.0.1 starting... Sat Aug 2 14:43:13 2008 #b7dea6c0: creating pidfile /var/run/gross/grossd.pid Sat Aug 2 14:43:13 2008 #b75ddb90: bloommgr starting... Sat Aug 2 14:43:13 2008 #b7dea6c0: Filters in sync. Starting... Sat Aug 2 14:43:13 2008 #b7dea6c0: starting postfix policy server Sat Aug 2 14:43:13 2008 #b5bdbb90: initializing postfix thread pool Sat Aug 2 14:43:13 2008 #b7dea6c0: starting milter policy server Sat Aug 2 14:44:14 2008 #b2bd5b90: a=match d=0 w=0 c=213.165.64.20 s= r= h=mail.gmx.net Sat Aug 2 14:48:14 2008 grossd status summary (begin, end, trust, match, greylist, block): 1217680993, 1217681294, 0, 1, 0, 0 Sat Aug 2 14:48:14 2008 grossd processing average delay (begin, end, trust[ms], match[ms], greylist[ms], block[ms]): 1217680993, 1217681294, 0.000, 0.000, 0.000, 0.000 Sat Aug 2 14:48:14 2008 grossd processing max delay (begin, end, trust[ms], match[ms], greylist[ms], block[ms]): 1217680993, 1217681294, 0.000, 0.000, 0.000, 0.000 Sat Aug 2 14:53:15 2008 grossd status summary (begin, end, trust, match, greylist, block): 1217681294, 1217681595, 0, 0, 0, 0 Sat Aug 2 14:53:15 2008 grossd processing average delay (begin, end, trust[ms], match[ms], greylist[ms], block[ms]): 1217681294, 1217681595, 0.000, 0.000, 0.000, 0.000 Sat Aug 2 14:53:15 2008 grossd processing max delay (begin, end, trust[ms], match[ms], greylist[ms], block[ms]): 1217681294, 1217681595, 0.000, 0.000, 0.000, 0.000 Sat Aug 2 14:58:16 2008 grossd status summary (begin, end, trust, match, greylist, block): 1217681595, 1217681896, 0, 0, 0, 0 Sat Aug 2 14:58:16 2008 grossd processing average delay (begin, end, trust[ms], match[ms], greylist[ms], block[ms]): 1217681595, 1217681896, 0.000, 0.000, 0.000, 0.000 Sat Aug 2 14:58:16 2008 grossd processing max delay (begin, end, trust[ms], match[ms], greylist[ms], block[ms]): 1217681595, 1217681896, 0.000, 0.000, 0.000, 0.000 Sat Aug 2 15:03:17 2008 grossd status summary (begin, end, trust, match, greylist, block): 1217681896, 1217682197, 0, 0, 0, 0 Sat Aug 2 15:03:17 2008 grossd processing average delay (begin, end, trust[ms], match[ms], greylist[ms], block[ms]): 1217681896, 1217682197, 0.000, 0.000, 0.000, 0.000 Sat Aug 2 15:03:17 2008 grossd processing max delay (begin, end, trust[ms], match[ms], greylist[ms], block[ms]): 1217681896, 1217682197, 0.000, 0.000, 0.000, 0.000 Sat Aug 2 15:08:18 2008 grossd status summary (begin, end, trust, match, greylist, block): 1217682197, 1217682498, 0, 0, 0, 0 Sat Aug 2 15:08:18 2008 grossd processing average delay (begin, end, trust[ms], match[ms], greylist[ms], block[ms]): 1217682197, 1217682498, 0.000, 0.000, 0.000, 0.000 Sat Aug 2 15:08:18 2008 grossd processing max delay (begin, end, trust[ms], match[ms], greylist[ms], block[ms]): 1217682197, 1217682498, 0.000, 0.000, 0.000, 0.000 Sat Aug 2 15:13:19 2008 grossd status summary (begin, end, trust, match, greylist, block): 1217682498, 1217682799, 0, 0, 0, 0 Sat Aug 2 15:13:19 2008 grossd processing average delay (begin, end, trust[ms], match[ms], greylist[ms], block[ms]): 1217682498, 1217682799, 0.000, 0.000, 0.000, 0.000 Sat Aug 2 15:13:19 2008 grossd processing max delay (begin, end, trust[ms], match[ms], greylist[ms], block[ms]): 1217682498, 1217682799, 0.000, 0.000, 0.000, 0.000 Sat Aug 2 15:18:20 2008 grossd status summary (begin, end, trust, match, greylist, block): 1217682799, 1217683100, 0, 0, 0, 0 Sat Aug 2 15:18:20 2008 grossd processing average delay (begin, end, trust[ms], match[ms], greylist[ms], block[ms]): 1217682799, 1217683100, 0.000, 0.000, 0.000, 0.000 Sat Aug 2 15:18:20 2008 grossd processing max delay (begin, end, trust[ms], match[ms], greylist[ms], block[ms]): 1217682799, 1217683100, 0.000, 0.000, 0.000, 0.000 Sat Aug 2 15:23:21 2008 grossd status summary (begin, end, trust, match, greylist, block): 1217683100, 1217683401, 0, 0, 0, 0 Sat Aug 2 15:23:21 2008 grossd processing average delay (begin, end, trust[ms], match[ms], greylist[ms], block[ms]): 1217683100, 1217683401, 0.000, 0.000, 0.000, 0.000 Sat Aug 2 15:23:21 2008 grossd processing max delay (begin, end, trust[ms], match[ms], greylist[ms], block[ms]): 1217683100, 1217683401, 0.000, 0.000, 0.000, 0.000 Sat Aug 2 15:23:51 2008 #b13d2b90: a=greylist d=17 w=0 c=66.150.96.100 s= r= h=smtp.emailenfuego.net Sat Aug 2 15:23:51 2008 #b23d4b90: a=greylist d=15 w=0 c=69.63.184.109 s= r= h=mx-out.facebook.com Sat Aug 2 15:24:18 2008 #b23d4b90: a=greylist d=0 w=0 c=83.9.74.216 s= r= h=acac216.neoplus.adsl.tpnet.pl Sat Aug 2 15:28:07 2008 #b75ddb90: received rotate command Sat Aug 2 15:28:07 2008 #b23d4b90: rotate thread starting Sat Aug 2 15:28:23 2008 grossd status summary (begin, end, trust, match, greylist, block): 1217683401, 1217683703, 0, 0, 3, 0 Sat Aug 2 15:28:23 2008 grossd processing average delay (begin, end, trust[ms], match[ms], greylist[ms], block[ms]): 1217683401, 1217683703, 0.000, 0.000, 10.667, 0.000 Sat Aug 2 15:28:23 2008 grossd processing max delay (begin, end, trust[ms], match[ms], greylist[ms], block[ms]): 1217683401, 1217683703, 0.000, 0.000, 17.000, 0.000 Sat Aug 2 15:29:16 2008 #b23d4b90: a=greylist d=0 w=0 c=192.168.0.135 s= r= h=pool-96-237-176-33.bstnma.fios.verizon.net [96.237.176.33] Sat Aug 2 15:30:36 2008 #b23d4b90: a=greylist d=0 w=0 c=69.63.184.102 s= r= h=mx-out.facebook.com Sat Aug 2 15:33:24 2008 grossd status summary (begin, end, trust, match, greylist, block): 1217683703, 1217684004, 0, 0, 2, 0 Sat Aug 2 15:33:24 2008 grossd processing average delay (begin, end, trust[ms], match[ms], greylist[ms], block[ms]): 1217683703, 1217684004, 0.000, 0.000, 0.000, 0.000 Sat Aug 2 15:33:24 2008 grossd processing max delay (begin, end, trust[ms], match[ms], greylist[ms], block[ms]): 1217683703, 1217684004, 0.000, 0.000, 17.000, 0.000 Sat Aug 2 15:35:26 2008 #b23d4b90: a=greylist d=0 w=0 c=208.124.238.82 s= r= h=[208.124.238.82] Sat Aug 2 15:37:46 2008 #b23d4b90: a=greylist d=0 w=0 c=192.168.0.135 s= r= h=[89.129.37.195] [89.129.37.195] Sat Aug 2 15:38:25 2008 grossd status summary (begin, end, trust, match, greylist, block): 1217684004, 1217684305, 0, 0, 2, 0 Sat Aug 2 15:38:25 2008 grossd processing average delay (begin, end, trust[ms], match[ms], greylist[ms], block[ms]): 1217684004, 1217684305, 0.000, 0.000, 0.000, 0.000 Sat Aug 2 15:38:25 2008 grossd processing max delay (begin, end, trust[ms], match[ms], greylist[ms], block[ms]): 1217684004, 1217684305, 0.000, 0.000, 17.000, 0.000 Segmentation fault mail ~ # > On what platform do you > run your email stack? > mail ~ # cat /etc/gentoo-release Gentoo Base System release 1.12.11.1 mail ~ # gcc -v Reading specs from /usr/lib/gcc/i686-pc-linux-gnu/3.4.6/specs Configured with: /var/tmp/portage/sys-devel/gcc-3.4.6-r2/work/gcc-3.4.6/configure --prefix=/usr --bindir=/usr/i686-pc-linux-gnu/gcc-bin/3.4.6 --includedir=/usr/lib/gcc/i686-pc-linux-gnu/3.4.6/include --datadir=/usr/share/gcc-data/i686-pc-linux-gnu/3.4.6 --mandir=/usr/share/gcc-data/i686-pc-linux-gnu/3.4.6/man --infodir=/usr/share/gcc-data/i686-pc-linux-gnu/3.4.6/info --with-gxx-include-dir=/usr/lib/gcc/i686-pc-linux-gnu/3.4.6/include/g++-v3 --host=i686-pc-linux-gnu --build=i686-pc-linux-gnu --disable-altivec --enable-nls --without-included-gettext --with-system-zlib --disable-checking --disable-werror --enable-secureplt --disable-libunwind-exceptions --disable-multilib --disable-libgcj --with-arch=i686 --enable-languages=c,c++,treelang --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu Thread model: posix gcc version 3.4.6 (Gentoo Hardened 3.4.6-r2 p1.5, ssp-3.4.6-1.0, pie-8.7.10) mail ~ # /lib/libc.so.6 GNU C Library stable release version 2.6.1, by Roland McGrath et al. Copyright (C) 2007 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Compiled by GNU CC version 3.4.6 (Gentoo Hardened 3.4.6-r2 p1.5, ssp-3.4.6-1.0, pie-8.7.10). Compiled on a Linux >>2.6.23.8<< system on 2008-06-23. Available extensions: C stubs add-on version 2.1.2 crypt add-on version 2.1 by Michael Glad and others Gentoo patchset 1.2 GNU Libidn by Simon Josefsson Native POSIX Threads Library by Ulrich Drepper et al Support for some architectures added on, not maintained in glibc core. BIND-8.2.3-T5B For bug reporting instructions, please see: . mail ~ # uname -a Linux mail 2.6.24.3 #1 Thu May 1 08:05:57 CEST 2008 i686 AMD Athlon (TM) AuthenticAMD GNU/Linux mail ~ # Soon I will try to run it here: nemesis ~ # cat /etc/gentoo-release Gentoo Base System release 2.0.0 nemesis ~ # gcc -v Using built-in specs. Target: i686-pc-linux-gnu Configured with: /var/tmp/portage/sys-devel/gcc-4.3.1-r1/work/gcc-4.3.1/configure --prefix=/usr --bindir=/usr/i686-pc-linux-gnu/gcc-bin/4.3.1 --includedir=/usr/lib/gcc/i686-pc-linux-gnu/4.3.1/include --datadir=/usr/share/gcc-data/i686-pc-linux-gnu/4.3.1 --mandir=/usr/share/gcc-data/i686-pc-linux-gnu/4.3.1/man --infodir=/usr/share/gcc-data/i686-pc-linux-gnu/4.3.1/info --with-gxx-include-dir=/usr/lib/gcc/i686-pc-linux-gnu/4.3.1/include/g++-v4 --host=i686-pc-linux-gnu --build=i686-pc-linux-gnu --disable-altivec --enable-nls --without-included-gettext --with-system-zlib --disable-checking --disable-werror --enable-secureplt --disable-multilib --enable-libmudflap --disable-libssp --enable-cld --disable-libgcj --with-arch=i686 --enable-languages=c,c++,treelang --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --with-bugurl=http://bugs.gentoo.org/ --with-pkgversion='Gentoo 4.3.1-r1 p1.1' Thread model: posix gcc version 4.3.1 (Gentoo 4.3.1-r1 p1.1) nemesis ~ # /lib/libc.so.6 GNU C Library stable release version 2.7, by Roland McGrath et al. Copyright (C) 2007 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Compiled by GNU CC version 4.3.1. Compiled on a Linux >>2.6.25.9<< system on 2008-07-06. Available extensions: C stubs add-on version 2.1.2 crypt add-on version 2.1 by Michael Glad and others Gentoo patchset 1.7 GNU Libidn by Simon Josefsson Native POSIX Threads Library by Ulrich Drepper et al Support for some architectures added on, not maintained in glibc core. BIND-8.2.3-T5B For bug reporting instructions, please see: . nemesis ~ # > > Now I have a question: How does gross 1.0.1 Milter knows that a mail > > is locally originating? I don't see any option in gross to disable it > > on mails originating from local addresses and/or I don't see any > > option in gross to disable it for authenticated users. With > > authenticated I mean users that succeeded SMTP AUTH ({auth_authen} > > Milter macro) and/or users that use STARTTLS with a verified client > > certificate ({verify} and {cert_subject} macro). Is there a way to > > bypass gross for those conditions? Does any one know that? > > I'm afraid that funcionality is not (yet) implemented in grossd. It > should be pretty straight forward to iplement those things in > worker_milter.c. > Okay. How do you handle that on your setup? You greylist everyone? Even your own users? > > What about white listing certain client IP's and/or recipient and/or > > senders? Is there a possibility to do that in gross (beside adding > > them in a local dnswl)? > > A local dnswl is the only option, currently. Senders could be > whitelisted with a local rhsbl with a negative weight. Now, I'm > uncertain if I have ever tested negative weights thorougly, but > It-Should-Work(TM). > Negative weights? Did not know that this works. Should be in the manual :) What about floats? Do they work as well? I mean 1.5 as weight. Would that work? What is the upper/lower bound of the weights? > > btw: I know, I know. I should/could use the Postfix policy with > > gross. But I just wanted to see if the Milter would work with > > Postfix. And it does :) But probably I will switch gross to use > > Postfix policy possibility. > > I'm going to investigate milter possibilities more after I've got our > JES to speak milter with gross. One interesting possibility would be to > extract URLs from message bodies and test them against Spamhaus SBL. > Uhh... If you talk about possibilities, then I can flood you with ideas: - Check SPF and allow me to act on the response. For example: - FAIL -> BLOCK/GREYLIST - PASS -> GREYLIST - SOFT FAIL -> GREYLIST - etc... the possibilities are endless :) - Check DKIM/SenderID - Check HELO/EHLO - the one claiming to be me (any of my domain) -> BLOCK/GREYLIST - the one having a non resolvable name -> BLOCK/GREYLIST - the one having an illegal HELO/EHLO -> GREYLIST - etc... - Check client name/ip - Greylist all matching certain REGEXP. For example: - /^(dhcp|dialup|cable|isdn|ppp|adsl|dsl|dial|dynamic|host|pool|port|ippool|static|user|pc|pop|client)[^\.]*[0-9]|^[^0-9]+[0-9]+\.[^\.]+\.adsl|^unknown$/i -> GREYLIST - /^[^\.]*[0-9][^0-9\.]+[0-9]|^[^\.]*[0-9]{5}|^([^\.]+\.)?[0-9][^\.]*\.[^\.]+\..+\.[a-z]|^[^\.]*[0-9]\.[^\.]*[0-9]-[0 -9]|^[^\.]*[0-9]\.[^\.]*[0-9]\.[^\.]+\..+\./i -> GREYLIST - etc... And, and, and, and... I think there are endless possibilities which could be added into gross. You even mentioned something about URLs in the body. Adding that kind of stuff would be very deluxe :) I don't know any greylisting solution out there doing body inspection in the greylisting phase. Gross would be the first one. > -- > Eino Tuominen > Steve -- GMX startet ShortView.de. Hier findest Du Leute mit Deinen Interessen! Jetzt dabei sein: http://www.shortview.de/wasistshortview.php?mc=sv_ext_mf at gmx From steeeeeveee at gmx.net Sat Aug 2 20:44:28 2008 From: steeeeeveee at gmx.net (Steve) Date: Sat, 02 Aug 2008 19:44:28 +0200 Subject: [Gross] gross 1.0.1 dnsbl/dnswl In-Reply-To: <4894964D.3060102@utu.fi> References: <20080802165922.149420@gmx.net> <4894964D.3060102@utu.fi> Message-ID: <20080802174428.149420@gmx.net> -------- Original-Nachricht -------- > Datum: Sat, 02 Aug 2008 20:15:57 +0300 > Von: Eino Tuominen > An: Steve > CC: gross at lists.utu.fi > Betreff: Re: [Gross] gross 1.0.1 dnsbl/dnswl > Steve wrote: > > Hello list > > > > Is there an possibility in gross to give different weight for a > > dnsbl/dnswl response? I mean: There are dnsbl's/dnswl's out there > > giving different results depending on the query you do. For example > > hostkarma.junkemailfilter.com sends following responses back: > > No, there is no way to do that. It would be an excellent enhancement, > though. Right now I can't come up with a clean way to express that in > config file. Maybe something like > > dnsbl=foo.org;5;127.0.0.1 > dnsbl=foo.org;3;127.0.0.2 > dnswl=foo.org;127.0.0.3 > Would not mind doing it that way as long as the lookups to foo.org would all be in one query. I mean that foo.org should be only queried once and the other requests for the result should be evaluated without again doing an lookup against foo.org. At least for the same connection/message. btw: What about timeouts to foo.org? How does gross handle them? Will gross continue to query foo.org even if foo.org does not respond? How long is the timeout? Is that configurable? Per query? Per query host? Will gross bark about it in the log if it fails to do an lookup? > The config system really needs a rewrite, the syntax is getting quite > patchy... > It works at least and it's not so ultra hard to get the configuration. I have seen worse things out there :) But yes. Having a cleaner syntax would help. > > Another example would be lookups to zz.countries.nerd.dk > > (http://countries.nerd.dk/more.html). They return the ISO 3166 A 2 > > country code of the IP. This is a great way to just make one single > > lookup and either white list all countries you don't want to greylist > > or to give with just one single lookup a weight to all the countries > > you would like to block/greylist. > > A good idea, too. > > -- > Eino Tuominen > Steve -- GMX startet ShortView.de. Hier findest Du Leute mit Deinen Interessen! Jetzt dabei sein: http://www.shortview.de/wasistshortview.php?mc=sv_ext_mf at gmx From jesse.thompson at doit.wisc.edu Mon Aug 4 21:48:44 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Mon, 04 Aug 2008 13:48:44 -0500 Subject: [Gross] gross 1.0.1 milter and Postfix In-Reply-To: <4894925A.2050101@utu.fi> References: <20080802164515.149420@gmx.net> <4894925A.2050101@utu.fi> Message-ID: <48974F0C.6070402@doit.wisc.edu> Eino Tuominen wrote: > Steve wrote: >> Now I have a question: How does gross 1.0.1 Milter knows that a mail >> is locally originating? I don't see any option in gross to disable it >> on mails originating from local addresses and/or I don't see any >> option in gross to disable it for authenticated users. With >> authenticated I mean users that succeeded SMTP AUTH ({auth_authen} >> Milter macro) and/or users that use STARTTLS with a verified client >> certificate ({verify} and {cert_subject} macro). Is there a way to >> bypass gross for those conditions? Does any one know that? > > I'm afraid that funcionality is not (yet) implemented in grossd. It > should be pretty straight forward to iplement those things in > worker_milter.c. Isn't this easier to implement in the MTA? Is there an advantage to doing this with gross? Maybe SJSMS channels make this easier than it would be on other MTAs. Jesse >> What about white listing certain client IP's and/or recipient and/or >> senders? Is there a possibility to do that in gross (beside adding >> them in a local dnswl)? > > A local dnswl is the only option, currently. Senders could be > whitelisted with a local rhsbl with a negative weight. Now, I'm > uncertain if I have ever tested negative weights thorougly, but > It-Should-Work(TM). > >> btw: I know, I know. I should/could use the Postfix policy with >> gross. But I just wanted to see if the Milter would work with >> Postfix. And it does :) But probably I will switch gross to use >> Postfix policy possibility. > > I'm going to investigate milter possibilities more after I've got our > JES to speak milter with gross. One interesting possibility would be to > extract URLs from message bodies and test them against Spamhaus SBL. > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3353 bytes Desc: S/MIME Cryptographic Signature URL: From steeeeeveee at gmx.net Mon Aug 4 22:14:03 2008 From: steeeeeveee at gmx.net (Steve) Date: Mon, 04 Aug 2008 21:14:03 +0200 Subject: [Gross] gross 1.0.1 milter and Postfix In-Reply-To: <48974F0C.6070402@doit.wisc.edu> References: <20080802164515.149420@gmx.net> <4894925A.2050101@utu.fi> <48974F0C.6070402@doit.wisc.edu> Message-ID: <20080804191403.259180@gmx.net> -------- Original-Nachricht -------- > Datum: Mon, 04 Aug 2008 13:48:44 -0500 > Von: Jesse Thompson > An: Eino Tuominen > CC: Steve , gross at lists.utu.fi > Betreff: Re: [Gross] gross 1.0.1 milter and Postfix > Eino Tuominen wrote: > > Steve wrote: > >> Now I have a question: How does gross 1.0.1 Milter knows that a mail > >> is locally originating? I don't see any option in gross to disable it > >> on mails originating from local addresses and/or I don't see any > >> option in gross to disable it for authenticated users. With > >> authenticated I mean users that succeeded SMTP AUTH ({auth_authen} > >> Milter macro) and/or users that use STARTTLS with a verified client > >> certificate ({verify} and {cert_subject} macro). Is there a way to > >> bypass gross for those conditions? Does any one know that? > > > > I'm afraid that funcionality is not (yet) implemented in grossd. It > > should be pretty straight forward to iplement those things in > > worker_milter.c. > > Isn't this easier to implement in the MTA? Is there an advantage to > doing this with gross? Maybe SJSMS channels make this easier than it > would be on other MTAs. > With the Postfix policy: Yes. It is very easy to implement that in the MTA level. With the Milter: Phuu... don't know how. The only thing I know is that I can say what macros to push to the Gross Milter but I am not aware of how to prevent a Milter from starting in Postfix. Maybe Sendmail has that kind of functionality? > Jesse > // Steve > > >> What about white listing certain client IP's and/or recipient and/or > >> senders? Is there a possibility to do that in gross (beside adding > >> them in a local dnswl)? > > > > A local dnswl is the only option, currently. Senders could be > > whitelisted with a local rhsbl with a negative weight. Now, I'm > > uncertain if I have ever tested negative weights thorougly, but > > It-Should-Work(TM). > > > >> btw: I know, I know. I should/could use the Postfix policy with > >> gross. But I just wanted to see if the Milter would work with > >> Postfix. And it does :) But probably I will switch gross to use > >> Postfix policy possibility. > > > > I'm going to investigate milter possibilities more after I've got our > > JES to speak milter with gross. One interesting possibility would be to > > extract URLs from message bodies and test them against Spamhaus SBL. > > > -- Psssst! Schon vom neuen GMX MultiMessenger geh?rt? Der kann`s mit allen: http://www.gmx.net/de/go/multimessenger From jesse.thompson at doit.wisc.edu Tue Aug 5 18:11:23 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Tue, 05 Aug 2008 10:11:23 -0500 Subject: [Gross] grossd problems after solaris 10 patching Message-ID: <48986D9B.3060601@doit.wisc.edu> After patching Solaris 10 on one server to the latest patch sets. We did not change gross on either host. Start grossd (-dDD) on the server that was patched. _STDOUT shows:_ Tue Aug 5 09:37:42 2008 #1: config: ... ... Tue Aug 5 09:37:42 2008 #1: Listening host address 1.2.3.4 Tue Aug 5 09:37:42 2008 #1: Sync listen address 1.2.3.4 Tue Aug 5 09:37:42 2008 #1: Peer 2.3.4.5 configured. Replicating. Tue Aug 5 09:37:42 2008 #1: Sync peer address 2.3.4.5 Tue Aug 5 09:37:42 2008 #1: updatestyle: GREY Tue Aug 5 09:37:42 2008 #1: adding dnsbl: ... ... Tue Aug 5 09:37:42 2008 #1: Blocker host address 127.0.0.1 Tue Aug 5 09:37:42 2008 #1: doubling the space for message queues from 1 to 2 Tue Aug 5 09:37:42 2008 #2: delay queue manager thread starting Tue Aug 5 09:37:42 2008 #2: waiting for messages Tue Aug 5 09:37:42 2008 #3: Found the correct state file magic string. Tue Aug 5 09:37:42 2008 #3: fixing bloom ring queue memory pointers, offset=0 Tue Aug 5 09:37:42 2008 #3: bloommgr starting... Tue Aug 5 09:37:42 2008 #4: Peer fd 6 Tue Aug 5 09:37:42 2008 #3: received rotate command Tue Aug 5 09:37:42 2008 #4: Waiting sync connection on host 144.92.197.229 port 1112 Tue Aug 5 09:37:42 2008 #6: rotate thread starting Tue Aug 5 09:37:42 2008 #6: rotation not needed (no activity for 10 minutes) _Truss shows minor activity_ /5: read(6, "\0\0\0\0\0\0\088", 8) = 8 /5: read(6, "\0\0\0\0\0\0 Q !\0\0\0\0".., 136) = 136 /5: clock_gettime(4, 0x00058814) = 0 /5: lwp_unpark(3) = 0 /3: lwp_park(0x00000000, 0) = 0 /3: time() = 1217947218 (similar activity is repeated in truss) (snoop indicates that the peer is seeing the activity) _The peer's log shows:_ Aug 5 09:37:42 grossd: [ID 702911 local3.info] #4: Got sync connection Aug 5 09:37:42 grossd: [ID 702911 local3.info] #4: Examining peer config Aug 5 09:37:42 grossd: [ID 702911 local3.error] #4: freeze queue 0 (continues to process client requests normally) _After the 10 minutes, the first gross server shows:_ Tue Aug 5 09:47:50 2008 #4: Got sync connection Tue Aug 5 09:47:50 2008 #4: Examining peer config (then nothing) (truss shows: lwp_park(0x00000000, 0) (sleeping...) ) _And the peer's log shows:_ Aug 5 09:47:50 grossd: [ID 702911 local3.error] #1299: Grossd shutdown with exit code 2: pthread_create Bad file number Aug 5 09:47:50 grossd: [ID 702911 local3.info] #3: bloommgr starting... Aug 5 09:47:50 grossd: [ID 702911 local3.info] #3: received rotate command Aug 5 09:47:50 grossd: [ID 702911 local3.info] #4: Waiting sync connection on host 2.3.4.5 port 1112 Aug 5 09:47:50 grossd: [ID 702911 local3.info] #6: rotate thread starting (no activity, including client requests, for the next X minutes) (truss shows: lwp_park(0x00000000, 0) (sleeping...) ) _So, I kill (-9 is required) gross on the first server, and the peer's log shows:_ Aug 5 09:54:16 grossd: [ID 702911 local3.info] #5: connection closed by client (still no activity in log or truss) I kill (-9 is required) gross on the peer, start is back up, and it is functioning fine as long as I don't start gross on the server that was just patched. I also tried deleting and recreating the state file on the upgraded server... same problem. _Also, I see from this morning, log entries like this, but I can't seem to trigger it now on demand:_ Aug 5 07:59:57 grossd: [ID 702911 local3.error] #4: Send filters: Bad file number Aug 5 07:59:57 last message repeated 291 times Aug 5 07:59:57 grossd: [ID 702911 local3.error] #4: Send filters: Bad file number Aug 5 07:59:57 last message repeated 402 times Aug 5 07:59:57 grossd: [ID 702911 local3.error] #4: Send filters: Bad file number Aug 5 07:59:57 last message repeated 21 times ... Any ideas? Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3353 bytes Desc: S/MIME Cryptographic Signature URL: From jesse.thompson at doit.wisc.edu Tue Aug 5 18:46:49 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Tue, 05 Aug 2008 10:46:49 -0500 Subject: [Gross] grossd problems after solaris 10 patching In-Reply-To: <48986D9B.3060601@doit.wisc.edu> References: <48986D9B.3060601@doit.wisc.edu> Message-ID: <489875E9.4030803@doit.wisc.edu> If I shut down the peer (with the older Solaris patch revs) and start up gross on the patched server, it processes client requests just fine. Then if I start up the peer, the same problems occurs (in reverse). We didn't see the problem in our test environment; most likely because we patched both of those servers at the same time. I presume that I could upgrade the other production server and this problem would go away... but it's hard to tell for sure. I changed the client port so that requests wouldn't come in, and I started gross on both servers with -dDD _Server 1 (started first):_ Tue Aug 5 10:24:33 2008 #3: bloommgr starting... Tue Aug 5 10:24:33 2008 #4: Connect to peer failed, host 1.2.3.4, port 1112 Tue Aug 5 10:24:33 2008 #4: Waiting sync connection on host 2.3.4.5 port 1112 Tue Aug 5 10:24:33 2008 #1: Filters in sync. Starting... Tue Aug 5 10:24:33 2008 #1: initializing dns checker thread pool 'dnsbl' Tue Aug 5 10:24:33 2008 #1: doubling the space for message queues from 2 to 4 Tue Aug 5 10:24:33 2008 #1: initializing Sophos blocker thread pool Tue Aug 5 10:24:33 2008 #5: threadpool 'dnsbl' thread #1 starting watchdog enabled Tue Aug 5 10:24:33 2008 #1: starting sjsms policy server Tue Aug 5 10:24:33 2008 #6: threadpool 'blocker' thread #1 starting watchdog enabled Tue Aug 5 10:24:33 2008 #7: initializing sjsms worker thread pool Tue Aug 5 10:24:33 2008 #7: doubling the space for message queues from 4 to 8 Tue Aug 5 10:24:33 2008 #9: threadpool 'sjsms' thread #1 starting Tue Aug 5 10:24:34 2008 #6: threadpool 'blocker' idling Tue Aug 5 10:24:34 2008 #5: threadpool 'dnsbl' idling Tue Aug 5 10:24:34 2008 #9: threadpool 'sjsms' idling etc... _Started Server 2:_ Tue Aug 5 10:24:41 2008 #3: bloommgr starting... Tue Aug 5 10:24:41 2008 #4: Peer fd 6 Tue Aug 5 10:24:41 2008 #3: received rotate command Tue Aug 5 10:24:41 2008 #4: Waiting sync connection on host 144.92.197.229 port 1112 Tue Aug 5 10:24:41 2008 #6: rotate thread starting Tue Aug 5 10:24:41 2008 #6: rotation not needed _Server 1 shows:_ Tue Aug 5 10:24:41 2008 #4: Got sync connection Tue Aug 5 10:24:41 2008 #4: Examining peer config Tue Aug 5 10:24:41 2008 #4: freeze queue 0 Tue Aug 5 10:24:42 2008 #5: threadpool 'dnsbl' idling Tue Aug 5 10:24:42 2008 #9: threadpool 'sjsms' idling Tue Aug 5 10:24:42 2008 #6: threadpool 'blocker' idling etc... Tue Aug 5 10:24:43 2008 #4: Sent buffer: 0 Tue Aug 5 10:24:44 2008 #5: threadpool 'dnsbl' idling Tue Aug 5 10:24:44 2008 #9: threadpool 'sjsms' idling Tue Aug 5 10:24:44 2008 #6: threadpool 'blocker' idling etc... Tue Aug 5 10:24:47 2008 #4: Sent buffer: 1 Tue Aug 5 10:24:48 2008 #6: threadpool 'blocker' idling Tue Aug 5 10:24:48 2008 #9: threadpool 'sjsms' idling Tue Aug 5 10:24:48 2008 #5: threadpool 'dnsbl' idling etc... Tue Aug 5 10:24:55 2008 #4: Sent buffer: 2 Tue Aug 5 10:24:56 2008 #9: threadpool 'sjsms' idling Tue Aug 5 10:24:56 2008 #5: threadpool 'dnsbl' idling Tue Aug 5 10:24:56 2008 #6: threadpool 'blocker' idling etc... Tue Aug 5 10:25:05 2008 #4: Sent buffer: 3 Tue Aug 5 10:25:06 2008 #6: threadpool 'blocker' idling Tue Aug 5 10:25:06 2008 #5: threadpool 'dnsbl' idling Tue Aug 5 10:25:06 2008 #9: threadpool 'sjsms' idling etc... It's hard to tell from the logs I pasted, since I snipped out most of the "idling" logs, but the time it takes for each buffer to be sent takes longer each time 1s, 4s, 8s, 10s Buffer #4 is never sent in the 20 minutes I was watching. I also did not see the "Bad file number" error occur. Jesse Jesse Thompson wrote: > After patching Solaris 10 on one server to the latest patch sets. We > did not change gross on either host. > > Start grossd (-dDD) on the server that was patched. > > _STDOUT shows:_ > Tue Aug 5 09:37:42 2008 #1: config: ... > ... > Tue Aug 5 09:37:42 2008 #1: Listening host address 1.2.3.4 > Tue Aug 5 09:37:42 2008 #1: Sync listen address 1.2.3.4 > Tue Aug 5 09:37:42 2008 #1: Peer 2.3.4.5 configured. Replicating. > Tue Aug 5 09:37:42 2008 #1: Sync peer address 2.3.4.5 > Tue Aug 5 09:37:42 2008 #1: updatestyle: GREY > Tue Aug 5 09:37:42 2008 #1: adding dnsbl: ... > ... > Tue Aug 5 09:37:42 2008 #1: Blocker host address 127.0.0.1 > Tue Aug 5 09:37:42 2008 #1: doubling the space for message queues from > 1 to 2 > Tue Aug 5 09:37:42 2008 #2: delay queue manager thread starting > Tue Aug 5 09:37:42 2008 #2: waiting for messages > Tue Aug 5 09:37:42 2008 #3: Found the correct state file magic string. > Tue Aug 5 09:37:42 2008 #3: fixing bloom ring queue memory pointers, > offset=0 > Tue Aug 5 09:37:42 2008 #3: bloommgr starting... > Tue Aug 5 09:37:42 2008 #4: Peer fd 6 > Tue Aug 5 09:37:42 2008 #3: received rotate command > Tue Aug 5 09:37:42 2008 #4: Waiting sync connection on host > 144.92.197.229 port 1112 > Tue Aug 5 09:37:42 2008 #6: rotate thread starting > Tue Aug 5 09:37:42 2008 #6: rotation not needed > > (no activity for 10 minutes) > > > _Truss shows minor activity_ > /5: read(6, "\0\0\0\0\0\0\088", 8) = 8 > /5: read(6, "\0\0\0\0\0\0 Q !\0\0\0\0".., 136) = 136 > /5: clock_gettime(4, 0x00058814) = 0 > /5: lwp_unpark(3) = 0 > /3: lwp_park(0x00000000, 0) = 0 > /3: time() = 1217947218 > > (similar activity is repeated in truss) > > (snoop indicates that the peer is seeing the activity) > > > _The peer's log shows:_ > Aug 5 09:37:42 grossd: [ID 702911 local3.info] #4: Got sync connection > Aug 5 09:37:42 grossd: [ID 702911 local3.info] #4: Examining peer config > Aug 5 09:37:42 grossd: [ID 702911 local3.error] #4: freeze queue 0 > > (continues to process client requests normally) > > > _After the 10 minutes, the first gross server shows:_ > Tue Aug 5 09:47:50 2008 #4: Got sync connection > Tue Aug 5 09:47:50 2008 #4: Examining peer config > > (then nothing) > > (truss shows: lwp_park(0x00000000, 0) (sleeping...) ) > > > _And the peer's log shows:_ > Aug 5 09:47:50 grossd: [ID 702911 local3.error] #1299: Grossd shutdown > with exit code 2: pthread_create Bad file number > Aug 5 09:47:50 grossd: [ID 702911 local3.info] #3: bloommgr starting... > Aug 5 09:47:50 grossd: [ID 702911 local3.info] #3: received rotate command > Aug 5 09:47:50 grossd: [ID 702911 local3.info] #4: Waiting sync > connection on host 2.3.4.5 port 1112 > Aug 5 09:47:50 grossd: [ID 702911 local3.info] #6: rotate thread starting > > (no activity, including client requests, for the next X minutes) > > (truss shows: lwp_park(0x00000000, 0) (sleeping...) ) > > > _So, I kill (-9 is required) gross on the first server, and the peer's > log shows:_ > Aug 5 09:54:16 grossd: [ID 702911 local3.info] #5: connection closed by > client > > (still no activity in log or truss) > > > I kill (-9 is required) gross on the peer, start is back up, and it is > functioning fine as long as I don't start gross on the server that was > just patched. > > > I also tried deleting and recreating the state file on the upgraded > server... same problem. > > > _Also, I see from this morning, log entries like this, but I can't seem > to trigger it now on demand:_ > Aug 5 07:59:57 grossd: [ID 702911 local3.error] #4: Send filters: Bad > file number > Aug 5 07:59:57 last message repeated 291 times > Aug 5 07:59:57 grossd: [ID 702911 local3.error] #4: Send filters: Bad > file number > Aug 5 07:59:57 last message repeated 402 times > Aug 5 07:59:57 grossd: [ID 702911 local3.error] #4: Send filters: Bad > file number > Aug 5 07:59:57 last message repeated 21 times > ... > > > Any ideas? > > Jesse > > > ------------------------------------------------------------------------ > > _______________________________________________ > Gross mailing list > Gross at lists.utu.fi > https://lists.utu.fi/mailman/listinfo/gross -- Jesse Thompson Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3353 bytes Desc: S/MIME Cryptographic Signature URL: From jesse.thompson at doit.wisc.edu Wed Aug 6 00:12:02 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Tue, 05 Aug 2008 16:12:02 -0500 Subject: [Gross] grossd problems after solaris 10 patching In-Reply-To: <489875E9.4030803@doit.wisc.edu> References: <48986D9B.3060601@doit.wisc.edu> <489875E9.4030803@doit.wisc.edu> Message-ID: <4898C222.2010906@doit.wisc.edu> We managed to get it to sync reliably if we move both zones onto the same server (patched OS). We can get it to work with one unpatched server and one patched server, but only under a specific scenario (see below). Given server A is not patched, and server B is patched. Scenario #1: Delete state files on both and recreate with -C. Start up server A; it runs normally. Start up server B. Server B has one thread that is effectively using an entire CPU. Server A's NLWP grows steadily until around 4000 (4096) and then crashes with the "Bad file number" error (trying to allocate the next thread.) Scenario #2: Delete state files on both and recreate with -C. Start up server B; it runs normally. Start up server A. Server A has one thread that is effectively using an entire CPU. Server B's NLWP grows steadily until around 450 and then server A finishes syncing and starts. Jesse Thompson wrote: > If I shut down the peer (with the older Solaris patch revs) and start up > gross on the patched server, it processes client requests just fine. > Then if I start up the peer, the same problems occurs (in reverse). > > We didn't see the problem in our test environment; most likely because > we patched both of those servers at the same time. I presume that I > could upgrade the other production server and this problem would go > away... but it's hard to tell for sure. > > > > I changed the client port so that requests wouldn't come in, and I > started gross on both servers with -dDD > > > _Server 1 (started first):_ > Tue Aug 5 10:24:33 2008 #3: bloommgr starting... > Tue Aug 5 10:24:33 2008 #4: Connect to peer failed, host 1.2.3.4, port 1112 > Tue Aug 5 10:24:33 2008 #4: Waiting sync connection on host 2.3.4.5 > port 1112 > Tue Aug 5 10:24:33 2008 #1: Filters in sync. Starting... > Tue Aug 5 10:24:33 2008 #1: initializing dns checker thread pool 'dnsbl' > Tue Aug 5 10:24:33 2008 #1: doubling the space for message queues from > 2 to 4 > Tue Aug 5 10:24:33 2008 #1: initializing Sophos blocker thread pool > Tue Aug 5 10:24:33 2008 #5: threadpool 'dnsbl' thread #1 starting > watchdog enabled > Tue Aug 5 10:24:33 2008 #1: starting sjsms policy server > Tue Aug 5 10:24:33 2008 #6: threadpool 'blocker' thread #1 starting > watchdog enabled > Tue Aug 5 10:24:33 2008 #7: initializing sjsms worker thread pool > Tue Aug 5 10:24:33 2008 #7: doubling the space for message queues from > 4 to 8 > Tue Aug 5 10:24:33 2008 #9: threadpool 'sjsms' thread #1 starting > Tue Aug 5 10:24:34 2008 #6: threadpool 'blocker' idling > Tue Aug 5 10:24:34 2008 #5: threadpool 'dnsbl' idling > Tue Aug 5 10:24:34 2008 #9: threadpool 'sjsms' idling > etc... > > > _Started Server 2:_ > Tue Aug 5 10:24:41 2008 #3: bloommgr starting... > Tue Aug 5 10:24:41 2008 #4: Peer fd 6 > Tue Aug 5 10:24:41 2008 #3: received rotate command > Tue Aug 5 10:24:41 2008 #4: Waiting sync connection on host > 144.92.197.229 port 1112 > Tue Aug 5 10:24:41 2008 #6: rotate thread starting > Tue Aug 5 10:24:41 2008 #6: rotation not needed > > > _Server 1 shows:_ > Tue Aug 5 10:24:41 2008 #4: Got sync connection > Tue Aug 5 10:24:41 2008 #4: Examining peer config > Tue Aug 5 10:24:41 2008 #4: freeze queue 0 > Tue Aug 5 10:24:42 2008 #5: threadpool 'dnsbl' idling > Tue Aug 5 10:24:42 2008 #9: threadpool 'sjsms' idling > Tue Aug 5 10:24:42 2008 #6: threadpool 'blocker' idling > etc... > Tue Aug 5 10:24:43 2008 #4: Sent buffer: 0 > Tue Aug 5 10:24:44 2008 #5: threadpool 'dnsbl' idling > Tue Aug 5 10:24:44 2008 #9: threadpool 'sjsms' idling > Tue Aug 5 10:24:44 2008 #6: threadpool 'blocker' idling > etc... > Tue Aug 5 10:24:47 2008 #4: Sent buffer: 1 > Tue Aug 5 10:24:48 2008 #6: threadpool 'blocker' idling > Tue Aug 5 10:24:48 2008 #9: threadpool 'sjsms' idling > Tue Aug 5 10:24:48 2008 #5: threadpool 'dnsbl' idling > etc... > Tue Aug 5 10:24:55 2008 #4: Sent buffer: 2 > Tue Aug 5 10:24:56 2008 #9: threadpool 'sjsms' idling > Tue Aug 5 10:24:56 2008 #5: threadpool 'dnsbl' idling > Tue Aug 5 10:24:56 2008 #6: threadpool 'blocker' idling > etc... > Tue Aug 5 10:25:05 2008 #4: Sent buffer: 3 > Tue Aug 5 10:25:06 2008 #6: threadpool 'blocker' idling > Tue Aug 5 10:25:06 2008 #5: threadpool 'dnsbl' idling > Tue Aug 5 10:25:06 2008 #9: threadpool 'sjsms' idling > etc... > > It's hard to tell from the logs I pasted, since I snipped out most of > the "idling" logs, but the time it takes for each buffer to be sent > takes longer each time > 1s, 4s, 8s, 10s > > Buffer #4 is never sent in the 20 minutes I was watching. I also did > not see the "Bad file number" error occur. > > Jesse > > > > > Jesse Thompson wrote: >> After patching Solaris 10 on one server to the latest patch sets. We >> did not change gross on either host. >> >> Start grossd (-dDD) on the server that was patched. >> >> _STDOUT shows:_ >> Tue Aug 5 09:37:42 2008 #1: config: ... >> ... >> Tue Aug 5 09:37:42 2008 #1: Listening host address 1.2.3.4 >> Tue Aug 5 09:37:42 2008 #1: Sync listen address 1.2.3.4 >> Tue Aug 5 09:37:42 2008 #1: Peer 2.3.4.5 configured. Replicating. >> Tue Aug 5 09:37:42 2008 #1: Sync peer address 2.3.4.5 >> Tue Aug 5 09:37:42 2008 #1: updatestyle: GREY >> Tue Aug 5 09:37:42 2008 #1: adding dnsbl: ... >> ... >> Tue Aug 5 09:37:42 2008 #1: Blocker host address 127.0.0.1 >> Tue Aug 5 09:37:42 2008 #1: doubling the space for message queues from >> 1 to 2 >> Tue Aug 5 09:37:42 2008 #2: delay queue manager thread starting >> Tue Aug 5 09:37:42 2008 #2: waiting for messages >> Tue Aug 5 09:37:42 2008 #3: Found the correct state file magic string. >> Tue Aug 5 09:37:42 2008 #3: fixing bloom ring queue memory pointers, >> offset=0 >> Tue Aug 5 09:37:42 2008 #3: bloommgr starting... >> Tue Aug 5 09:37:42 2008 #4: Peer fd 6 >> Tue Aug 5 09:37:42 2008 #3: received rotate command >> Tue Aug 5 09:37:42 2008 #4: Waiting sync connection on host >> 144.92.197.229 port 1112 >> Tue Aug 5 09:37:42 2008 #6: rotate thread starting >> Tue Aug 5 09:37:42 2008 #6: rotation not needed >> >> (no activity for 10 minutes) >> >> >> _Truss shows minor activity_ >> /5: read(6, "\0\0\0\0\0\0\088", 8) = 8 >> /5: read(6, "\0\0\0\0\0\0 Q !\0\0\0\0".., 136) = 136 >> /5: clock_gettime(4, 0x00058814) = 0 >> /5: lwp_unpark(3) = 0 >> /3: lwp_park(0x00000000, 0) = 0 >> /3: time() = 1217947218 >> >> (similar activity is repeated in truss) >> >> (snoop indicates that the peer is seeing the activity) >> >> >> _The peer's log shows:_ >> Aug 5 09:37:42 grossd: [ID 702911 local3.info] #4: Got sync connection >> Aug 5 09:37:42 grossd: [ID 702911 local3.info] #4: Examining peer config >> Aug 5 09:37:42 grossd: [ID 702911 local3.error] #4: freeze queue 0 >> >> (continues to process client requests normally) >> >> >> _After the 10 minutes, the first gross server shows:_ >> Tue Aug 5 09:47:50 2008 #4: Got sync connection >> Tue Aug 5 09:47:50 2008 #4: Examining peer config >> >> (then nothing) >> >> (truss shows: lwp_park(0x00000000, 0) (sleeping...) ) >> >> >> _And the peer's log shows:_ >> Aug 5 09:47:50 grossd: [ID 702911 local3.error] #1299: Grossd shutdown >> with exit code 2: pthread_create Bad file number >> Aug 5 09:47:50 grossd: [ID 702911 local3.info] #3: bloommgr starting... >> Aug 5 09:47:50 grossd: [ID 702911 local3.info] #3: received rotate command >> Aug 5 09:47:50 grossd: [ID 702911 local3.info] #4: Waiting sync >> connection on host 2.3.4.5 port 1112 >> Aug 5 09:47:50 grossd: [ID 702911 local3.info] #6: rotate thread starting >> >> (no activity, including client requests, for the next X minutes) >> >> (truss shows: lwp_park(0x00000000, 0) (sleeping...) ) >> >> >> _So, I kill (-9 is required) gross on the first server, and the peer's >> log shows:_ >> Aug 5 09:54:16 grossd: [ID 702911 local3.info] #5: connection closed by >> client >> >> (still no activity in log or truss) >> >> >> I kill (-9 is required) gross on the peer, start is back up, and it is >> functioning fine as long as I don't start gross on the server that was >> just patched. >> >> >> I also tried deleting and recreating the state file on the upgraded >> server... same problem. >> >> >> _Also, I see from this morning, log entries like this, but I can't seem >> to trigger it now on demand:_ >> Aug 5 07:59:57 grossd: [ID 702911 local3.error] #4: Send filters: Bad >> file number >> Aug 5 07:59:57 last message repeated 291 times >> Aug 5 07:59:57 grossd: [ID 702911 local3.error] #4: Send filters: Bad >> file number >> Aug 5 07:59:57 last message repeated 402 times >> Aug 5 07:59:57 grossd: [ID 702911 local3.error] #4: Send filters: Bad >> file number >> Aug 5 07:59:57 last message repeated 21 times >> ... >> >> >> Any ideas? >> >> Jesse >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Gross mailing list >> Gross at lists.utu.fi >> https://lists.utu.fi/mailman/listinfo/gross > > > ------------------------------------------------------------------------ > > _______________________________________________ > Gross mailing list > Gross at lists.utu.fi > https://lists.utu.fi/mailman/listinfo/gross -- Jesse Thompson Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3353 bytes Desc: S/MIME Cryptographic Signature URL: From natanael.copa at gmail.com Thu Sep 11 16:48:51 2008 From: natanael.copa at gmail.com (Natanael Copa) Date: Thu, 11 Sep 2008 15:48:51 +0200 Subject: [Gross] [PATCH] option -u user Message-ID: <1221140931.16944.66.camel@nc.nor.wtbts.org> Hi, Attatched patch will let you chose what user you want run your deamon. Defaults to 'nobody' so it should be backwards compatible. thanks! -nc -------------- next part -------------- A non-text attachment was scrubbed... Name: gross-user.patch Type: text/x-patch Size: 2548 bytes Desc: not available URL: From eino at utu.fi Fri Sep 12 12:27:51 2008 From: eino at utu.fi (Eino Tuominen) Date: Fri, 12 Sep 2008 12:27:51 +0300 Subject: [Gross] [PATCH] option -u user In-Reply-To: <1221140931.16944.66.camel@nc.nor.wtbts.org> References: <1221140931.16944.66.camel@nc.nor.wtbts.org> Message-ID: <48CA3617.107@utu.fi> Natanael Copa wrote: > Hi, > > Attatched patch will let you chose what user you want run your deamon. > Defaults to 'nobody' so it should be backwards compatible. Thanks! I committed it to 1.0 branch and trunk. I made you a Gross project member on googlecode, so you have now commit privileges on svn. Welcome! I also noticed that Google has made some improvements to the project hosting system. I made a new RFE template that should enable you to post new issues as RFE:s instead (or in addition) of bugs. Also, there's new code review feature in source code browsing tool, now open for everyone. -- Eino Tuominen From Jarno.Huuskonen at uku.fi Mon Sep 29 11:59:33 2008 From: Jarno.Huuskonen at uku.fi (Jarno Huuskonen) Date: Mon, 29 Sep 2008 11:59:33 +0300 Subject: [Gross] gross-1.0.1 milter and rhsbl ? Message-ID: <20080929085933.GF2653@laivuri133.uku.fi> Hello, I noticed small problem how grossd+milter handles email addresses and rhsbl: - if the smtp client sends addresses like: , then grossd parses the domain: example.com> and queries rhsbl lists: example.com>.rhsbl.sorbs.net This probably doesn't happen w/postfix: If I understand the protocol description (http://www.postfix.org/SMTPD_POLICY_README.html) postfix sends the domain/address w/out <> ? Perhaps grossd should in (check_dnsbl.c:dnsblc) check if the sender domain ends with '>' --> and remove the trailing '>' ? Also, something like this might give a better error message when the config file is not readable: diff -urN gross-1.0.1.orig/src/conf.c gross-1.0.1/src/conf.c --- gross-1.0.1.orig/src/conf.c 2008-05-15 17:21:50.000000000 +0300 +++ gross-1.0.1/src/conf.c 2008-09-26 18:32:33.000000000 +0300 @@ -323,8 +323,10 @@ /* open configfile for reading */ fd = open(filename, O_RDONLY); - if (fd < 0) + if (fd < 0) { + fprintf(stderr, "Can't open configfile: %s\n", filename); return config; + } /* * Process the config file. -Jarno -- Jarno Huuskonen From eino at utu.fi Mon Sep 29 12:13:08 2008 From: eino at utu.fi (Eino Tuominen) Date: Mon, 29 Sep 2008 12:13:08 +0300 Subject: [Gross] gross-1.0.1 milter and rhsbl ? In-Reply-To: <20080929085933.GF2653@laivuri133.uku.fi> References: <20080929085933.GF2653@laivuri133.uku.fi> Message-ID: <48E09C24.6050201@utu.fi> Jarno Huuskonen wrote: > > Perhaps grossd should in (check_dnsbl.c:dnsblc) check if the sender > domain ends with '>' --> and remove the trailing '>' ? Or, maybe it would be better if '<' and '>' were dropped in worker_milter.c? RHSBL has been very inefficient everywhere it has been tested. Have you found a good and efficient list? > Also, something like this might give a better error message when > the config file is not readable: Thanks, -- Eino Tuominen From Jarno.Huuskonen at uku.fi Mon Sep 29 15:12:11 2008 From: Jarno.Huuskonen at uku.fi (Jarno Huuskonen) Date: Mon, 29 Sep 2008 15:12:11 +0300 Subject: [Gross] gross-1.0.1 milter and rhsbl ? In-Reply-To: <48E09C24.6050201@utu.fi> References: <20080929085933.GF2653@laivuri133.uku.fi> <48E09C24.6050201@utu.fi> Message-ID: <20080929121211.GK2653@laivuri133.uku.fi> Hi, On Mon, Sep 29, Eino Tuominen wrote: > Jarno Huuskonen wrote: > > > > Perhaps grossd should in (check_dnsbl.c:dnsblc) check if the sender > > domain ends with '>' --> and remove the trailing '>' ? > > Or, maybe it would be better if '<' and '>' were dropped in worker_milter.c? Yes, that's probably a better idea especially if postfix already sends addresses w/out '<>'. > RHSBL has been very inefficient everywhere it has been tested. Have you > found a good and efficient list? No I haven't, just testing with the sorbs list. -Jarno -- Jarno Huuskonen From eino at utu.fi Tue Oct 21 11:45:43 2008 From: eino at utu.fi (Eino Tuominen) Date: Tue, 21 Oct 2008 11:45:43 +0300 Subject: [Gross] Skipping authenticated users (was: gross 1.0.1 milter and Postfix) In-Reply-To: <20080802164515.149420@gmx.net> References: <20080802164515.149420@gmx.net> Message-ID: <48FD96B7.8030707@utu.fi> Steve wrote: > > authenticated I mean users that succeeded SMTP AUTH ({auth_authen} > Milter macro) and/or users that use STARTTLS with a verified client > certificate ({verify} and {cert_subject} macro). Is there a way to > bypass gross for those conditions? Does any one know that? Diggin in the pile of things to do... I have now implemented a change in worker_milter.c to skip authenticated users. I can't find any value of filtering those in gross, so I ask you all if this feature should be made configurable? -- Eino Tuominen From eino at utu.fi Tue Oct 21 12:22:54 2008 From: eino at utu.fi (Eino Tuominen) Date: Tue, 21 Oct 2008 12:22:54 +0300 Subject: [Gross] Patch to Greylist Bad Helo In-Reply-To: <696c56870804300449r19d63c29u56987e14856acae9@mail.gmail.com> References: <696c56870804300449r19d63c29u56987e14856acae9@mail.gmail.com> Message-ID: <48FD9F6E.7050608@utu.fi> Jeff Chan wrote: > Hi, > > For these few days, I went over the top and ran an experiement to > greylist senders with bad helo strings using grossd. I did this > because I found many obvious spams slipped through the RBLs undetected > (as trusted). Those were the ones with bad helo strings, either with > non-ascii characters, illegal domain name characters, non-fully > qualified hostnames, or with IP addresses whether bracketed or not. Hi Jeff, Sorry for a very late response... I have now incorporated your piece of code in the new check_helo.c in svn version. This will be relased in version 1.1, which I plan to release next (with the 1.0.2 patch release). -- Eino Tuominen From R.E.Sonneveld at sonnection.nl Tue Oct 21 12:34:49 2008 From: R.E.Sonneveld at sonnection.nl (Rolf E. Sonneveld) Date: Tue, 21 Oct 2008 11:34:49 +0200 Subject: [Gross] Patch to Greylist Bad Helo In-Reply-To: <48FD9F6E.7050608@utu.fi> References: <696c56870804300449r19d63c29u56987e14856acae9@mail.gmail.com> <48FD9F6E.7050608@utu.fi> Message-ID: <48FDA239.7020605@sonnection.nl> Eino Tuominen wrote: > Jeff Chan wrote: > >> Hi, >> >> For these few days, I went over the top and ran an experiement to >> greylist senders with bad helo strings using grossd. I did this >> because I found many obvious spams slipped through the RBLs undetected >> (as trusted). Those were the ones with bad helo strings, either with >> non-ascii characters, illegal domain name characters, non-fully >> qualified hostnames, or with IP addresses whether bracketed or not. >> > > Hi Jeff, > > Sorry for a very late response... > > I have now incorporated your piece of code in the new check_helo.c in > svn version. > > This will be relased in version 1.1, which I plan to release next (with > the 1.0.2 patch release). > Is this configurable? What is the logic? I assume something like: if ( on DNSBL) or (we_greylist_everything turned on) then perform greylist action else if then perform greylist action else OK fi fi Or? /rolf -------------- next part -------------- An HTML attachment was scrubbed... URL: From cite at incertum.net Tue Oct 21 14:23:33 2008 From: cite at incertum.net (Stefan =?utf-8?Q?F=C3=B6rster?=) Date: Tue, 21 Oct 2008 13:23:33 +0200 Subject: [Gross] Skipping authenticated users In-Reply-To: <48FD96B7.8030707@utu.fi> References: <20080802164515.149420@gmx.net> <48FD96B7.8030707@utu.fi> Message-ID: <20081021112333.GH3476@mail.incertum.net> * Eino Tuominen wrote: > Steve wrote: >> authenticated I mean users that succeeded SMTP AUTH ({auth_authen} >> Milter macro) and/or users that use STARTTLS with a verified client >> certificate ({verify} and {cert_subject} macro). Is there a way to >> bypass gross for those conditions? Does any one know that? > > Diggin in the pile of things to do... > > > I have now implemented a change in worker_milter.c to skip authenticated > users. I can't find any value of filtering those in gross, so I ask you > all if this feature should be made configurable? With Postfix, this is a non-issue - simply add "permit_sasl_authenticated" or "permit_tls_clientcerts" in front of "check_policy_service". Cheers Stefan -- Stefan F?rster http://www.incertum.net/ Public Key: 0xBBE2A9E9 From eino at utu.fi Tue Oct 21 14:38:12 2008 From: eino at utu.fi (Eino Tuominen) Date: Tue, 21 Oct 2008 14:38:12 +0300 Subject: [Gross] Patch to Greylist Bad Helo In-Reply-To: <48FDA239.7020605@sonnection.nl> References: <696c56870804300449r19d63c29u56987e14856acae9@mail.gmail.com> <48FD9F6E.7050608@utu.fi> <48FDA239.7020605@sonnection.nl> Message-ID: <48FDBF24.8040104@utu.fi> Rolf E. Sonneveld wrote: > > Is this configurable? > What is the logic? I assume something like: It's a separate check that needs to be enabled. (check = helo) Currently (not yet committed to the svn) the weights are as follows: weight = 0 if (helo string is invalid) { weight += 1 } else if (helo string does not resolve) { weight += 1 } else if (helo string resolves but does not match client ip) { weight += 1 } if (client ip has no PTR record) { weight += 1 } else if (helo string and the PTR for client ip do not match) { weight += 1 } I'll add configurable weights before releasing it. -- Eino Tuominen From eino at utu.fi Tue Oct 21 14:43:05 2008 From: eino at utu.fi (Eino Tuominen) Date: Tue, 21 Oct 2008 14:43:05 +0300 Subject: [Gross] Skipping authenticated users In-Reply-To: <20081021112333.GH3476@mail.incertum.net> References: <20080802164515.149420@gmx.net> <48FD96B7.8030707@utu.fi> <20081021112333.GH3476@mail.incertum.net> Message-ID: <48FDC049.8090102@utu.fi> Stefan F?rster wrote: >> >> I have now implemented a change in worker_milter.c to skip authenticated >> users. I can't find any value of filtering those in gross, so I ask you >> all if this feature should be made configurable? > > With Postfix, this is a non-issue - simply add > "permit_sasl_authenticated" or "permit_tls_clientcerts" in front of > "check_policy_service". But what if you configure Postfix to use grossd as a milter? -- Eino Tuominen From cite at incertum.net Tue Oct 21 14:49:15 2008 From: cite at incertum.net (Stefan =?utf-8?Q?F=C3=B6rster?=) Date: Tue, 21 Oct 2008 13:49:15 +0200 Subject: [Gross] Skipping authenticated users In-Reply-To: <48FDC049.8090102@utu.fi> References: <20080802164515.149420@gmx.net> <48FD96B7.8030707@utu.fi> <20081021112333.GH3476@mail.incertum.net> <48FDC049.8090102@utu.fi> Message-ID: <20081021114915.GI3476@mail.incertum.net> * Eino Tuominen wrote: > Stefan F?rster wrote: >>> >>> I have now implemented a change in worker_milter.c to skip authenticated >>> users. I can't find any value of filtering those in gross, so I ask you >>> all if this feature should be made configurable? >> >> With Postfix, this is a non-issue - simply add >> "permit_sasl_authenticated" or "permit_tls_clientcerts" in front of >> "check_policy_service". > > But what if you configure Postfix to use grossd as a milter? It becomes an issue, then ;-) Cheers Stefan -- Stefan F?rster http://www.incertum.net/ Public Key: 0xBBE2A9E9 From steeeeeveee at gmx.net Tue Oct 21 21:29:40 2008 From: steeeeeveee at gmx.net (Steve) Date: Tue, 21 Oct 2008 20:29:40 +0200 Subject: [Gross] Skipping authenticated users In-Reply-To: <20081021112333.GH3476@mail.incertum.net> References: <20080802164515.149420@gmx.net> <48FD96B7.8030707@utu.fi> <20081021112333.GH3476@mail.incertum.net> Message-ID: <20081021182940.99740@gmx.net> -------- Original-Nachricht -------- > Datum: Tue, 21 Oct 2008 13:23:33 +0200 > Von: "Stefan F?rster" > An: gross at lists.utu.fi > Betreff: Re: [Gross] Skipping authenticated users > * Eino Tuominen wrote: > > Steve wrote: > >> authenticated I mean users that succeeded SMTP AUTH ({auth_authen} > >> Milter macro) and/or users that use STARTTLS with a verified client > >> certificate ({verify} and {cert_subject} macro). Is there a way to > >> bypass gross for those conditions? Does any one know that? > > > > Diggin in the pile of things to do... > > > > > > I have now implemented a change in worker_milter.c to skip authenticated > > users. I can't find any value of filtering those in gross, so I ask you > > all if this feature should be made configurable? > > With Postfix, this is a non-issue - simply add > "permit_sasl_authenticated" or "permit_tls_clientcerts" in front of > "check_policy_service". > And how would you do that with Postfix when using Gross as a Milter? > > Cheers > Stefan > -- > Stefan F?rster http://www.incertum.net/ Public Key: 0xBBE2A9E9 > _______________________________________________ > Gross mailing list > Gross at lists.utu.fi > https://lists.utu.fi/mailman/listinfo/gross -- Psssst! Schon vom neuen GMX MultiMessenger geh?rt? Der kann`s mit allen: http://www.gmx.net/de/go/multimessenger From jpiszcz at lucidpixels.com Sun Oct 26 02:12:39 2008 From: jpiszcz at lucidpixels.com (Justin Piszcz) Date: Sat, 25 Oct 2008 19:12:39 -0400 (EDT) Subject: [Gross] Excellent program [ feature requests ] Message-ID: For RHSBL checks, can we implement the following: - client rhsbl checks (hostname of client) - helo rhsbl checks (what the client says the helo=) - sender rhsbl checks (I assume this is what it does by default) Similar to the following in postfix: - reject_rhsbl_helo - reject_rhsbl_client # 'block_threshold' is the threshold after which grossd sends # a permanent error to the client. Every check that considers client_ip # as suspicious returns a value (check weight). When sum of these # values gets equivalent or greater than 'block_threshold', grossd # sends a STATUS_BLOCK response. Default is 0, which disables # this functionality. # DEFAULT: block_threshold = 0 block_threshold = 4 - Would it be possible to block if in more than X number of RBLs as well, regardless of the score? From jpiszcz at lucidpixels.com Sun Oct 26 02:16:09 2008 From: jpiszcz at lucidpixels.com (Justin Piszcz) Date: Sat, 25 Oct 2008 19:16:09 -0400 (EDT) Subject: [Gross] Excellent program [ feature requests ] In-Reply-To: References: Message-ID: On Sat, 25 Oct 2008, Justin Piszcz wrote: > For RHSBL checks, can we implement the following: > - client rhsbl checks (hostname of client) > - helo rhsbl checks (what the client says the helo=) > - sender rhsbl checks (I assume this is what it does by default) > > Similar to the following in postfix: > - reject_rhsbl_helo > - reject_rhsbl_client > > # 'block_threshold' is the threshold after which grossd sends > # a permanent error to the client. Every check that considers client_ip > # as suspicious returns a value (check weight). When sum of these > # values gets equivalent or greater than 'block_threshold', grossd > # sends a STATUS_BLOCK response. Default is 0, which disables > # this functionality. > # DEFAULT: block_threshold = 0 > block_threshold = 4 > - Would it be possible to block if in more than X number of RBLs as well, > regardless of the score? > > Also could be a combination of RBL+RHSBL hits. Oct 25 19:14:38 p34 postfix/smtpd[16101]: connect from unknown[123.69.100.47] Oct 25 19:14:40 p34 grossd: #4123f950: a=block d=114 w=4 c=123.69.100.47 s=dyr57yur54f at huashanfu.com r=jpiszcz at lucidpixels.com h=huashanfu.com m=cbl.abuseat.org+1 m=b.barracudacentral.org+3 Would it be possible also to set the prefix to postfix/grossd or have it user-customizable without having to change the source? Its nice to have the output line-up :) Justin. From eino at utu.fi Sun Oct 26 09:29:53 2008 From: eino at utu.fi (Eino Tuominen) Date: Sun, 26 Oct 2008 09:29:53 +0200 Subject: [Gross] Excellent program [ feature requests ] In-Reply-To: References: Message-ID: <49041C71.7070804@utu.fi> Justin Piszcz wrote: > For RHSBL checks, can we implement the following: > - client rhsbl checks (hostname of client) > - helo rhsbl checks (what the client says the helo=) > - sender rhsbl checks (I assume this is what it does by default) > > Similar to the following in postfix: > - reject_rhsbl_helo > - reject_rhsbl_client Sure, that's possible. But, are there any rhsbl's that truly work efficiently? > # DEFAULT: block_threshold = 0 > block_threshold = 4 > - Would it be possible to block if in more than X number of RBLs as well, > regardless of the score? Yes it would (technically), but do you think it would really help? I mean, could you just set up the weights for the RBL checks? Future considerations: After the new configuration system it would be more easy to define check groups(, or check clusters). I mean one could state something like this in the config file: check spamhaus { type = dnsbl zone = zen.spamhaus.org in 127.0.0.4/30 { weight = 4 } in 127.0.0.2/32 { weight = 8 } weight = 1 } check sorbs { type = dnsbl zone = dnsbl.sorbs.net delay = 200 ms } check spamcop { type = dnsbl zone = bl.spamcop.net weight = 2 } # if matches both spamhaus and spamcop, adjust weight cluster hauscop { matches = {spamhaus, spamcop} weight = 4 } I'm not sure if this would help grossd to be more efficient, but at least it's doable. Also, the config format above is far from complete. I want it to be understandable without the manual in hand. New ideas and contributions are always welcome. -- Eino Tuominen From jpiszcz at lucidpixels.com Sun Oct 26 11:24:19 2008 From: jpiszcz at lucidpixels.com (Justin Piszcz) Date: Sun, 26 Oct 2008 05:24:19 -0400 (EDT) Subject: [Gross] Gross not greylisting in postfix properly [?] Message-ID: Current greylisting with gross: Oct 26 04:37:28 p34 postfix/smtpd[11478]: connect from dslb-084-061-200-033.pools.arcor-ip.net[84.61.200.33] Oct 26 04:37:28 p34 grossd: #40fdb950: a=greylist d=123 w=3 c=84.61.200.33 s=nicotine at herbalwerks.com r=user at destination.com h=xppe.arcor-ip.net m=zen.spamhaus.org+3 [ .. ] a later policy/restriction takes over instead [ .. ] Oct 26 04:37:28 p34 postfix/geoip[11481]: address[84.61.200.33] country[DE, Germany] result[strictcheckslvl2] Oct 26 04:37:28 p34 postfix/smtpd[11478]: NOQUEUE: reject: RCPT from dslb-084-061-200-033.pools.arcor-ip.net[84.61.200.33]: 454 4.7.1 Service unavailable; Client host [84.61.200.33] blocked using dul.dnsbl.sorbs.net; Dynamic IP Addresses See: http://www.sorbs.net/lookup.shtml?84.61.200.33; from= to= proto=SMTP helo= Oct 26 04:37:28 p34 postfix/smtpd[11478]: disconnect from dslb-084-061-200-033.pools.arcor-ip.net[84.61.200.33] To fix this same problem in postgrey, one modifies the POSTGREY_OPTS from: (Default = DEFER_IF_PERMIT) POSTGREY_OPTS="--inet=127.0.0.1:60000" To: POSTGREY_OPTS="--inet=127.0.0.1:60000 --greylist-action=421" So when it gets to the specific restriction in postfix [grossd] it will greylist and not continue to the next check. Is this possible to get this added into grossd or make the action specifiable? So grossd greylists the e-mail first before it gets to the next check? ------ So I disabled all other policies in postfix to test the greylisting feature in grossd and disabled the BLOCK in grossd (which works well): Oct 26 05:02:31 p34 postfix/smtpd[25864]: connect from dslb-084-061-200-033.pools.arcor-ip.net[84.61.200.33] Oct 26 05:02:31 p34 grossd: #40fdb950: a=match d=0 w=0 c=84.61.200.33 s=nicotine at herbalwerks.com r=user at destination.com h=xppe.arcor-ip.net Oct 26 05:02:31 p34 postfix/smtpd[25864]: 857323288A: client=dslb-084-061-200-033.pools.arcor-ip.net[84.61.200.33] Oct 26 05:02:31 p34 postfix/cleanup[25868]: 857323288A: message-id=<1421119838.20081026090052 at jawaclub.com> Oct 26 05:02:32 p34 postfix/qmgr[24721]: 857323288A: from=, size=4220, nrcpt=1 (queue active) Oct 26 05:02:32 p34 postfix/smtpd[25864]: disconnect from dslb-084-061-200-033.pools.arcor-ip.net[84.61.200.33] Oct 26 05:02:33 p34 postfix/smtpd[25872]: connect from localhost.localdomain[127.0.0.1] Oct 26 05:02:33 p34 postfix/smtpd[25872]: 58D0A41E473: client=localhost.localdomain[127.0.0.1] Oct 26 05:02:33 p34 postfix/cleanup[25868]: 58D0A41E473: message-id=<1421119838.20081026090052 at jawaclub.com> Oct 26 05:02:33 p34 postfix/smtpd[25872]: disconnect from localhost.localdomain[127.0.0.1] Oct 26 05:02:33 p34 postfix/qmgr[24721]: 58D0A41E473: from=, size=4912, nrcpt=1 (queue active) Oct 26 05:02:33 p34 amavis[922]: (00922-06) Passed SPAM, [84.61.200.33] [84.61.200.33] -> , Message-ID: <1421119838.20081026090052 at jawaclub.com>, mail_id: PjrnJnmguvMK, Hits: 3.884, size: 4220, queued_as: 58D0A41E473, 1363 ms Oct 26 05:02:33 p34 postfix/lmtp[25869]: 857323288A: to=, relay=127.0.0.1[127.0.0.1]:10024, delay=2, delays=0.68/0/0/1.4, dsn=2.0.0, status=sent (250 2.0.0 Ok, id=00922-06, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 58D0A41E473) Oct 26 05:02:33 p34 postfix/qmgr[24721]: 857323288A: removed Oct 26 05:02:34 p34 postfix/local[25873]: 58D0A41E473: to=, relay=local, delay=1.1, delays=0.02/0/0/1.1, dsn=2.0.0, status=sent (delivered to command: procmail -a "$EXTENSION") Oct 26 05:02:34 p34 postfix/qmgr[24721]: 58D0A41E473: removed If you check the IP: 84.61.200.33 It is listed in zen.spamhaus.org: dnsbl = zen.spamhaus.org;3 The threshold: grey_threshold = 1 And block_threshold is default (0). Why did this get let through and not greylisted instead? Justin. From jpiszcz at lucidpixels.com Sun Oct 26 11:28:31 2008 From: jpiszcz at lucidpixels.com (Justin Piszcz) Date: Sun, 26 Oct 2008 05:28:31 -0400 (EDT) Subject: [Gross] Excellent program [ feature requests ] In-Reply-To: <49041C71.7070804@utu.fi> References: <49041C71.7070804@utu.fi> Message-ID: On Sun, 26 Oct 2008, Eino Tuominen wrote: > Justin Piszcz wrote: >> For RHSBL checks, can we implement the following: >> - client rhsbl checks (hostname of client) >> - helo rhsbl checks (what the client says the helo=) >> - sender rhsbl checks (I assume this is what it does by default) >> >> Similar to the following in postfix: >> - reject_rhsbl_helo >> - reject_rhsbl_client > > Sure, that's possible. But, are there any rhsbl's that truly work > efficiently? Yes: black.uribl.com, multi.surbl.org, multi.uribl.com, rhsbl.ahbl.org > >> # DEFAULT: block_threshold = 0 >> block_threshold = 4 >> - Would it be possible to block if in more than X number of RBLs as well, >> regardless of the score? > > Yes it would (technically), but do you think it would really help? I mean, > could you just set up the weights for the RBL checks? Yes for instance the weights could be different but if you have 10 RBLs and its blocked in 5, the number of RBLs supersedes the weights. Such functionality is implemented in the (unmaintained) policyd-weight, www.policyd-weight.org. Also, policyd-weight adds a PREPEND to the header of the e-mail to show you what the filter did (policyd-weight)- would this be possible to add to gross as well? > > > Future considerations: > > After the new configuration system it would be more easy to define check > groups(, or check clusters). I mean one could state something like this in > the config file: > > check spamhaus { > type = dnsbl > zone = zen.spamhaus.org > in 127.0.0.4/30 { > weight = 4 > } > in 127.0.0.2/32 { > weight = 8 > } > weight = 1 > } > > check sorbs { > type = dnsbl > zone = dnsbl.sorbs.net > delay = 200 ms > } > > check spamcop { > type = dnsbl > zone = bl.spamcop.net > weight = 2 > } > > # if matches both spamhaus and spamcop, adjust weight > cluster hauscop { > matches = {spamhaus, spamcop} > weight = 4 > } > > I'm not sure if this would help grossd to be more efficient, but at least > it's doable. > > Also, the config format above is far from complete. I want it to be > understandable without the manual in hand. New ideas and contributions are > always welcome. > > -- > Eino Tuominen > From cite at incertum.net Sun Oct 26 11:32:06 2008 From: cite at incertum.net (Stefan =?utf-8?Q?F=C3=B6rster?=) Date: Sun, 26 Oct 2008 10:32:06 +0100 Subject: [Gross] Gross not greylisting in postfix properly [?] In-Reply-To: References: Message-ID: <20081026093205.GA23086@mail.incertum.net> * Justin Piszcz wrote: > Current greylisting with gross: > > Oct 26 04:37:28 p34 postfix/smtpd[11478]: connect from dslb-084-061-200-033.pools.arcor-ip.net[84.61.200.33] > Oct 26 04:37:28 p34 grossd: #40fdb950: a=greylist d=123 w=3 c=84.61.200.33 s=nicotine at herbalwerks.com r=user at destination.com h=xppe.arcor-ip.net m=zen.spamhaus.org+3 > > [ .. ] a later policy/restriction takes over instead [ .. ] > > Oct 26 04:37:28 p34 postfix/geoip[11481]: address[84.61.200.33] country[DE, Germany] result[strictcheckslvl2] > Oct 26 04:37:28 p34 postfix/smtpd[11478]: NOQUEUE: reject: RCPT from dslb-084-061-200-033.pools.arcor-ip.net[84.61.200.33]: 454 4.7.1 Service unavailable; Client host [84.61.200.33] blocked using dul.dnsbl.sorbs.net; Dynamic IP Addresses See: http://www.sorbs.net/lookup.shtml?84.61.200.33; from= to= proto=SMTP helo= > Oct 26 04:37:28 p34 postfix/smtpd[11478]: disconnect from dslb-084-061-200-033.pools.arcor-ip.net[84.61.200.33] > > To fix this same problem in postgrey, one modifies the POSTGREY_OPTS from: > > (Default = DEFER_IF_PERMIT) > > POSTGREY_OPTS="--inet=127.0.0.1:60000" > > To: > > POSTGREY_OPTS="--inet=127.0.0.1:60000 --greylist-action=421" > > So when it gets to the specific restriction in postfix [grossd] it will > greylist and not continue to the next check. > > Is this possible to get this added into grossd or make the action specifiable? > So grossd greylists the e-mail first before it gets to the next check? Set "postfix_response_grey" accordingly (available in version 1.0.1). Cheers Stefan -- Stefan F?rster http://www.incertum.net/ Public Key: 0xBBE2A9E9 From cite at incertum.net Sun Oct 26 11:39:54 2008 From: cite at incertum.net (Stefan =?utf-8?Q?F=C3=B6rster?=) Date: Sun, 26 Oct 2008 10:39:54 +0100 Subject: [Gross] Excellent program [ feature requests ] In-Reply-To: References: Message-ID: <20081026093953.GB23086@mail.incertum.net> * Justin Piszcz wrote: > For RHSBL checks, can we implement the following: > - client rhsbl checks (hostname of client) > - helo rhsbl checks (what the client says the helo=) > - sender rhsbl checks (I assume this is what it does by default) > > Similar to the following in postfix: > - reject_rhsbl_helo > - reject_rhsbl_client > > # 'block_threshold' is the threshold after which grossd sends > # a permanent error to the client. Every check that considers client_ip > # as suspicious returns a value (check weight). When sum of these > # values gets equivalent or greater than 'block_threshold', grossd > # sends a STATUS_BLOCK response. Default is 0, which disables > # this functionality. > # DEFAULT: block_threshold = 0 > block_threshold = 4 > - Would it be possible to block if in more than X number of RBLs as well, > regardless of the score? I was not confident that the additional logic implemented in grossd was a good thing[tm] when grossd came out first: Grossd is a fantastic greylisting software, and as such, development efforts should go into the two things that grossd really excels at: Greylisting and replicating the database - and only those two. For things like scoring/weighting hits in DNSBLs, doing HELO/EHLO checks and so on, there are programs available which already have the functionality to do this: policyd (v2), postfwd and policyd-weight come to my mind. Especially postfwd with it's semi Turing-complete configuration language is suited very well to return an action like "greylisting" and therefore invoke the named smtpd_restriction_class. Bloating grossd with the kind of stuff you are suggesting above is not a good idea, IMHO. Unix philsophy is "One tool, one job" - and that's for a _reason_. Now this said, there might be reasons to do this for the people running grossd with Sendmail, SMSJ or other MTAs. For Postfix, all of this is completely unnecessary and better left to tools explicitely designed for the task. Ciao Stefan -- Stefan F?rster http://www.incertum.net/ Public Key: 0xBBE2A9E9 "Vor allem brauchste nen richtig derben Anmachspruch, z.B.: 'Hey, schonmal in's Koma gev?gelt worden?' Nur so hinterl?sst Du Eindruck!" -- Frank From cite at incertum.net Sun Oct 26 11:44:11 2008 From: cite at incertum.net (Stefan =?utf-8?Q?F=C3=B6rster?=) Date: Sun, 26 Oct 2008 10:44:11 +0100 Subject: [Gross] Skipping authenticated users In-Reply-To: <20081021182940.99740@gmx.net> References: <20080802164515.149420@gmx.net> <48FD96B7.8030707@utu.fi> <20081021112333.GH3476@mail.incertum.net> <20081021182940.99740@gmx.net> Message-ID: <20081026094411.GC23086@mail.incertum.net> * Steve wrote: >> With Postfix, this is a non-issue - simply add >> "permit_sasl_authenticated" or "permit_tls_clientcerts" in front of >> "check_policy_service". >> > And how would you do that with Postfix when using Gross as a Milter? Why on earth would anyone do that? Disclaimer: I don't know much about the milter interface - it _might_ be that you really have to implement that logic in grossd if the MTAs relaying on the use of milters don't support ordered restrictions before they call the milter. Ciao Stefan -- Stefan F?rster http://www.incertum.net/ Public Key: 0xBBE2A9E9 There's little time to analyze data in the field. If it moves, shoot it. If it doesn't move, it's probably faking, shoot it. If it's unarmed and surrendering, shoot it twice. From jpiszcz at lucidpixels.com Sun Oct 26 14:07:38 2008 From: jpiszcz at lucidpixels.com (Justin Piszcz) Date: Sun, 26 Oct 2008 08:07:38 -0400 (EDT) Subject: [Gross] Gross 1.0.1 gotchas (statefile + postfix greylist/syntax) Message-ID: 1. By default the statefile directory may not exist on your system and there are no errors in the logs to indicate why greylisting isn't working. - Read manual first of course :) - Specify the statefile, make sure permissions are correct. - Create the statefile grossd -C Add to init.d script: DAEMON=/app/gross-1.0.1-x86_64/sbin/grossd STATEDIR=/var/lib/grossd STATEFILE="$STATEDIR/grossd.state" if [ ! -d "$STATEDIR" ]; then echo "grossd: State directory not found, creating..." mkdir -p "$STATEDIR" chown nobody "$STATEDIR" fi if [ ! -f "$STATEFILE" ]; then echo "grossd: Statefile not found, creating..." $DAEMON -C fi 2. Postfix not rejecting the e-mail and using DEFER_IF_PERMIT instead. - The proper 'syntax' to fix that is shown below: # Use postfix greylisting. postfix_response_grey = action=421 Greylisted, please try again later Maybe include this in the FAQ or README (the example) of how it works with postfix. I tried: postfix_response_grey = 421 (postfix errored on this one) postfix_response_grey = action="421 Greylisted" (also errored) postfix_response_grey = action = 421 Greylisted.. etc (correct) Justin. From jesse.thompson at doit.wisc.edu Mon Oct 27 21:36:54 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Mon, 27 Oct 2008 14:36:54 -0500 Subject: [Gross] Excellent program [ feature requests ] In-Reply-To: <20081026093953.GB23086@mail.incertum.net> References: <20081026093953.GB23086@mail.incertum.net> Message-ID: <49061856.7020300@doit.wisc.edu> Stefan F?rster wrote: > * Justin Piszcz wrote: >> For RHSBL checks, can we implement the following: >> - client rhsbl checks (hostname of client) >> - helo rhsbl checks (what the client says the helo=) >> - sender rhsbl checks (I assume this is what it does by default) >> >> Similar to the following in postfix: >> - reject_rhsbl_helo >> - reject_rhsbl_client >> >> # 'block_threshold' is the threshold after which grossd sends >> # a permanent error to the client. Every check that considers client_ip >> # as suspicious returns a value (check weight). When sum of these >> # values gets equivalent or greater than 'block_threshold', grossd >> # sends a STATUS_BLOCK response. Default is 0, which disables >> # this functionality. >> # DEFAULT: block_threshold = 0 >> block_threshold = 4 >> - Would it be possible to block if in more than X number of RBLs as well, >> regardless of the score? > > I was not confident that the additional logic implemented in grossd > was a good thing[tm] when grossd came out first: Grossd is a fantastic > greylisting software, and as such, development efforts should go into > the two things that grossd really excels at: Greylisting and > replicating the database - and only those two. > > For things like scoring/weighting hits in DNSBLs, doing HELO/EHLO > checks and so on, there are programs available which already have the > functionality to do this: policyd (v2), postfwd and policyd-weight > come to my mind. Especially postfwd with it's semi Turing-complete > configuration language is suited very well to return an action like > "greylisting" and therefore invoke the named smtpd_restriction_class. > Bloating grossd with the kind of stuff you are suggesting above is not > a good idea, IMHO. Unix philsophy is "One tool, one job" - and that's > for a _reason_. > > Now this said, there might be reasons to do this for the people > running grossd with Sendmail, SMSJ or other MTAs. For Postfix, all of > this is completely unnecessary and better left to tools explicitely > designed for the task. If you take away the "scoring/weighting hits in DNSBLs, doing HELO/EHLO checks and so on" GROSS would be only GRO. The other programs you mention might be perfectly capable, but weighted/conditional blacklisting/greylisting works better if you can use multiple criteria in the calculation. So, Gross would be better if it either implements some of those features, or comes up with a way to integrate with these programs. Jesse -- Jesse Thompson Division of Information Technology, University of Wisconsin-Madison Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3353 bytes Desc: S/MIME Cryptographic Signature URL: From jpiszcz at lucidpixels.com Fri Nov 14 22:08:12 2008 From: jpiszcz at lucidpixels.com (Justin Piszcz) Date: Fri, 14 Nov 2008 15:08:12 -0500 (EST) Subject: [Gross] gross logging question Message-ID: In my grossd.conf: # EXAMPLE: stat_type = status # EXAMPLE: stat_type = delay stat_type = none # 'stat_interval' is the number of seconds between status log entries # DEFAULT: stat_interval = 3600 stat_interval = 0 ------ However, in the postfix logs, I am still seeing this: Nov 14 13:52:52 l1 grossd: #4296a950: received rotate command Nov 14 13:52:52 l1 grossd: #46171950: rotate thread starting ------- I want to see what gross's actions are but I do not wish to see it rotating etc, how do I disable that? Justin. From jesse.thompson at doit.wisc.edu Tue Nov 18 23:07:16 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Tue, 18 Nov 2008 15:07:16 -0600 Subject: [Gross] gross logging question In-Reply-To: References: Message-ID: <49232E84.9020104@doit.wisc.edu> Well, the stat_type and stat_interval options have nothing to do with the log verbosity. The default logging level is the least verbose, so IIRC the only way to avoid getting those logs in the default logging is to modify the code before compiling. i.e. change those log command from LOG_INFO to LOG_DEBUG. This should be easy for you to do, even if you aren't a coder. Without thinking about it in depth, I can't think why those two log entries really need to be in the default logging level. Perhaps it's intentional, or it could be a regression. Eino probably has more to say about this. Jesse Justin Piszcz wrote: > In my grossd.conf: > > # EXAMPLE: stat_type = status > # EXAMPLE: stat_type = delay > stat_type = none > > # 'stat_interval' is the number of seconds between status log entries > # DEFAULT: stat_interval = 3600 > stat_interval = 0 > > ------ > > However, in the postfix logs, I am still seeing this: > > Nov 14 13:52:52 l1 grossd: #4296a950: received rotate command > Nov 14 13:52:52 l1 grossd: #46171950: rotate thread starting > > > ------- > > I want to see what gross's actions are but I do not wish to see it > rotating etc, how do I disable that? > > Justin. > _______________________________________________ > Gross mailing list > Gross at lists.utu.fi > https://lists.utu.fi/mailman/listinfo/gross -- Jesse Thompson Division of Information Technology, University of Wisconsin-Madison Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3353 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Wed Nov 19 08:28:42 2008 From: eino at utu.fi (Eino Tuominen) Date: Wed, 19 Nov 2008 08:28:42 +0200 Subject: [Gross] gross logging question In-Reply-To: <49232E84.9020104@doit.wisc.edu> References: <49232E84.9020104@doit.wisc.edu> Message-ID: <4923B21A.8030601@utu.fi> Jesse Thompson wrote: > Well, the stat_type and stat_interval options have nothing to do with > the log verbosity. > > The default logging level is the least verbose, so IIRC the only way to > avoid getting those logs in the default logging is to modify the code > before compiling. i.e. change those log command from LOG_INFO to > LOG_DEBUG. This should be easy for you to do, even if you aren't a coder. > > Without thinking about it in depth, I can't think why those two log > entries really need to be in the default logging level. Perhaps it's > intentional, or it could be a regression. > > Eino probably has more to say about this. Yes, there indeed are some log entries still using wrong log levels. Those should be using LOG_DEBUG instead of LOG_INFO. -- Eino Tuominen From jesse.thompson at doit.wisc.edu Wed Nov 19 17:43:13 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Wed, 19 Nov 2008 09:43:13 -0600 Subject: [Gross] gross logging question In-Reply-To: <4923B21A.8030601@utu.fi> References: <49232E84.9020104@doit.wisc.edu> <4923B21A.8030601@utu.fi> Message-ID: <49243411.1000004@doit.wisc.edu> Eino Tuominen wrote: > Jesse Thompson wrote: >> Well, the stat_type and stat_interval options have nothing to do with >> the log verbosity. >> >> The default logging level is the least verbose, so IIRC the only way to >> avoid getting those logs in the default logging is to modify the code >> before compiling. i.e. change those log command from LOG_INFO to >> LOG_DEBUG. This should be easy for you to do, even if you aren't a >> coder. >> >> Without thinking about it in depth, I can't think why those two log >> entries really need to be in the default logging level. Perhaps it's >> intentional, or it could be a regression. >> >> Eino probably has more to say about this. > > Yes, there indeed are some log entries still using wrong log levels. > Those should be using LOG_DEBUG instead of LOG_INFO. I committed the change to svn. Jesse -- Jesse Thompson Division of Information Technology, University of Wisconsin-Madison Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3353 bytes Desc: S/MIME Cryptographic Signature URL: From jpiszcz at lucidpixels.com Wed Nov 19 17:57:17 2008 From: jpiszcz at lucidpixels.com (Justin Piszcz) Date: Wed, 19 Nov 2008 10:57:17 -0500 (EST) Subject: [Gross] gross logging question In-Reply-To: <49243411.1000004@doit.wisc.edu> References: <49232E84.9020104@doit.wisc.edu> <4923B21A.8030601@utu.fi> <49243411.1000004@doit.wisc.edu> Message-ID: On Wed, 19 Nov 2008, Jesse Thompson wrote: > Eino Tuominen wrote: >> Jesse Thompson wrote: >>> Well, the stat_type and stat_interval options have nothing to do with >>> the log verbosity. >>> >>> The default logging level is the least verbose, so IIRC the only way to >>> avoid getting those logs in the default logging is to modify the code >>> before compiling. i.e. change those log command from LOG_INFO to >>> LOG_DEBUG. This should be easy for you to do, even if you aren't a >>> coder. >>> >>> Without thinking about it in depth, I can't think why those two log >>> entries really need to be in the default logging level. Perhaps it's >>> intentional, or it could be a regression. >>> >>> Eino probably has more to say about this. >> >> Yes, there indeed are some log entries still using wrong log levels. >> Those should be using LOG_DEBUG instead of LOG_INFO. > > I committed the change to svn. > > Jesse New version built and installed, will let you know if any issues, thanks! Justin. From eino at utu.fi Wed Nov 19 18:50:54 2008 From: eino at utu.fi (Eino Tuominen) Date: Wed, 19 Nov 2008 18:50:54 +0200 Subject: [Gross] gross logging question In-Reply-To: <49243411.1000004@doit.wisc.edu> References: <49232E84.9020104@doit.wisc.edu> <4923B21A.8030601@utu.fi> <49243411.1000004@doit.wisc.edu> Message-ID: <492443EE.6010107@utu.fi> Jesse Thompson wrote: > > I committed the change to svn. Thanks, I merged this to 1.0 branch, too. -- Eino Tuominen From eino at utu.fi Wed Nov 19 18:51:00 2008 From: eino at utu.fi (Eino Tuominen) Date: Wed, 19 Nov 2008 18:51:00 +0200 Subject: [Gross] Gross sources in svn Message-ID: <492443F4.6060503@utu.fi> Hi, Just a reminder: the sources from the svn trunk are not of production quality. If you wish to run the latest stable version of grossd, please use 1.0 branch: svn checkout http://gross.googlecode.com/svn/trunk/ gross-1.0 -- Eino Tuominen From steeeeeveee at gmx.net Tue Dec 9 13:23:37 2008 From: steeeeeveee at gmx.net (Steve) Date: Tue, 09 Dec 2008 12:23:37 +0100 Subject: [Gross] Can't run configure on system with glibc-2.7 Message-ID: <20081209112337.266310@gmx.net> Hello list I can't run configure on a system with glibc 2.7 installed on it. The error I get is: ---- tisiphone ~ # ./configure --prefix=/usr --host=i686-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --sysconfdir=/etc/mail/gross --includedir=/usr/include --enable-milter --enable-dnsbl --build=i686-pc-linux-gnu checking build system type... i686-pc-linux-gnu checking host system type... i686-pc-linux-gnu checking target system type... i686-pc-linux-gnu checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking for i686-pc-linux-gnu-cc... no checking for i686-pc-linux-gnu-gcc... i686-pc-linux-gnu-gcc checking for C compiler default output file name... a.out checking whether the C compiler works... yes checking whether we are cross compiling... no checking for suffix of executables... checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether i686-pc-linux-gnu-gcc accepts -g... yes checking for i686-pc-linux-gnu-gcc option to accept ISO C89... none needed checking for style of include used by make... GNU checking dependency style of i686-pc-linux-gnu-gcc... gcc3 checking for a BSD-compatible install... /usr/bin/install -c checking for a sed that does not truncate output... /bin/sed checking for grep that handles long lines and -e... /bin/grep checking for egrep... /bin/grep -E checking for ld used by i686-pc-linux-gnu-gcc... /usr/i686-pc-linux-gnu/bin/ld checking if the linker (/usr/i686-pc-linux-gnu/bin/ld) is GNU ld... yes checking for /usr/i686-pc-linux-gnu/bin/ld option to reload object files... -r checking for BSD-compatible nm... /usr/bin/nm -B checking whether ln -s works... yes checking how to recognise dependent libraries... pass_all checking how to run the C preprocessor... i686-pc-linux-gnu-gcc -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking dlfcn.h usability... yes checking dlfcn.h presence... yes checking for dlfcn.h... yes checking the maximum length of command line arguments... 32768 checking command to parse /usr/bin/nm -B output from i686-pc-linux-gnu-gcc object... ok checking for objdir... .libs checking for i686-pc-linux-gnu-ar... i686-pc-linux-gnu-ar checking for i686-pc-linux-gnu-ranlib... i686-pc-linux-gnu-ranlib checking for i686-pc-linux-gnu-strip... i686-pc-linux-gnu-strip checking if i686-pc-linux-gnu-gcc supports -fno-rtti -fno-exceptions... no checking for i686-pc-linux-gnu-gcc option to produce PIC... -fPIC checking if i686-pc-linux-gnu-gcc PIC flag -fPIC works... yes checking if i686-pc-linux-gnu-gcc static flag -static works... yes checking if i686-pc-linux-gnu-gcc supports -c -o file.o... yes checking whether the i686-pc-linux-gnu-gcc linker (/usr/i686-pc-linux-gnu/bin/ld) supports shared libraries... yes checking whether -lc should be explicitly linked in... no checking dynamic linker characteristics... GNU/Linux ld.so checking how to hardcode library paths into programs... immediate checking whether stripping libraries is possible... yes checking if libtool supports shared libraries... yes checking whether to build shared libraries... yes checking whether to build static libraries... yes configure: creating libtool checking for library containing sem_init... no configure: error: Can't compile without semaphores. tisiphone ~ # ---- But everyting is there: ---- tisiphone ~ # grep -R "sem_init" /usr/include/ /usr/include/semaphore.h:extern int sem_init (sem_t *__sem, int __pshared, unsigned int __value) tisiphone ~ # ls -lah /usr/include/semaphore.h -rw-r--r-- 1 root root 2.5K Dec 7 13:20 /usr/include/semaphore.h tisiphone ~ # ---- The other tools I use are: ---- tisiphone ~ # /lib/libpthread.so.0 Native POSIX Threads Library by Ulrich Drepper et al Copyright (C) 2006 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Forced unwind support included. tisiphone ~ # /lib/libc.so.6 GNU C Library stable release version 2.7, by Roland McGrath et al. Copyright (C) 2007 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Compiled by GNU CC version 4.3.2. Compiled on a Linux >>2.6.28-rc7<< system on 2008-12-07. Available extensions: C stubs add-on version 2.1.2 crypt add-on version 2.1 by Michael Glad and others Gentoo patchset 1.7 GNU Libidn by Simon Josefsson Native POSIX Threads Library by Ulrich Drepper et al Support for some architectures added on, not maintained in glibc core. BIND-8.2.3-T5B For bug reporting instructions, please see: . tisiphone ~ # gcc --version gcc (Gentoo 4.3.2 p1.2) 4.3.2 Copyright (C) 2008 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. tisiphone ~ # libtool --version ltmain.sh (GNU libtool) 2.2.4 Written by Gordon Matzigkeit , 1996 Copyright (C) 2008 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. tisiphone ~ # emerge --info Portage 2.2_rc17 (default/linux/x86/2008.0/server, gcc-4.3.2, glibc-2.7-r2, 2.6.28-rc7 i686) ================================================================= System uname: Linux-2.6.28-rc7-i686-Intel-R-_Pentium-R-_4_CPU_2.80GHz-with-glibc2.0 Timestamp of tree: Mon, 08 Dec 2008 00:30:01 +0000 app-shells/bash: 3.2_p33 dev-lang/python: 2.5.2-r7 sys-apps/baselayout: 2.0.0 sys-apps/openrc: 0.3.0-r1 sys-apps/sandbox: 1.2.18.1-r2 sys-devel/autoconf: 2.61-r2 sys-devel/automake: 1.5, 1.9.6-r2, 1.10.1-r1 sys-devel/binutils: 2.18-r3 sys-devel/gcc-config: 1.4.0-r4 sys-devel/libtool: 2.2.4 virtual/os-headers: 2.6.27-r2 ACCEPT_KEYWORDS="x86" CBUILD="i686-pc-linux-gnu" CFLAGS="-march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /var/bind" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/eselect/postgresql /etc/gconf /etc/gentoo-release /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/terminfo /etc/udev/rules.d" CXXFLAGS="-march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow -fno-enforce-eh-specs" DISTDIR="/usr/portage/distfiles" FEATURES="distlocks fixpackages parallel-fetch preserve-libs protect-owned sandbox sfperms strict unmerge-orphans userfetch" GENTOO_MIRRORS="ftp://sunsite.cnlab-switch.ch/mirror/gentoo http://www.ibiblio.org/pub/Linux/distributions/gentoo http://gentoo.oregonstate.edu/" LDFLAGS="-Wl,-O1 -Wl,--add-needed -Wl,--as-needed -Wl,--hash-style=both -Wl,--sort-common" MAKEOPTS="-j3" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/mnt/gentoo.overlay" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="7zip acl addition aio ares bcmath bzip2 calendar cgi cli cracklib crypt ctype cups curl curlwrappers daemon diffheaders dlz dovecot-sasl dri exif fastcgi force-cgi-redirect fortran ftp gd gd-external gmp gnutls hash http iconv idn imap isdnlog ithreads jpeg json large-domain logrotate mailwrapper mbox md5sum mhash midi milter mktemp mmap mmx mudflap multiuser ncurses network-cron nls nptl nptlonly openmp pcntl pcre pdo perl png pop posix postgres pppd python readline reflection resolvconf rpc session sharedext sharedmem simplexml smtp snmp soap sockets spell spl sse sse2 ssl status sub subversion suhosin symlink sysfs syslog sysvipc tcpd threads tidy tokenizer truetype unicode urandom vda vhosts virtual-users webdav x86 xattr xml xmlreader xmlrpc xmlwriter xorg xsl xslt zip zip-external zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via8 Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LANG, LC_ALL, LINGUAS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS tisiphone ~ # ---- What am I doing wrong? // Steve -- Psssst! Schon vom neuen GMX MultiMessenger geh?rt? Der kann`s mit allen: http://www.gmx.net/de/go/multimessenger From eino at utu.fi Tue Dec 9 13:33:39 2008 From: eino at utu.fi (Eino Tuominen) Date: Tue, 09 Dec 2008 13:33:39 +0200 Subject: [Gross] Can't run configure on system with glibc-2.7 In-Reply-To: <20081209112337.266310@gmx.net> References: <20081209112337.266310@gmx.net> Message-ID: <493E5793.7060409@utu.fi> Steve wrote: > > checking for library containing sem_init... no > configure: error: Can't compile without semaphores. > tisiphone ~ # > ---- > > But everyting is there: > ---- > tisiphone ~ # grep -R "sem_init" /usr/include/ > /usr/include/semaphore.h:extern int sem_init (sem_t *__sem, int __pshared, unsigned int __value) > tisiphone ~ # ls -lah /usr/include/semaphore.h > -rw-r--r-- 1 root root 2.5K Dec 7 13:20 /usr/include/semaphore.h > tisiphone ~ # > ---- Hi, Check config.log first. There you should find the test program configure is trying to compile. It may be that configure is unable to find the library function for some reason. -- Eino Tuominen From steeeeeveee at gmx.net Tue Dec 9 14:22:56 2008 From: steeeeeveee at gmx.net (Steve) Date: Tue, 09 Dec 2008 13:22:56 +0100 Subject: [Gross] Can't run configure on system with glibc-2.7 In-Reply-To: <493E5793.7060409@utu.fi> References: <20081209112337.266310@gmx.net> <493E5793.7060409@utu.fi> Message-ID: <20081209122256.266320@gmx.net> Here the full config log: ----- tisiphone ~ # cat /var/tmp/portage/mail-filter/gross-1.0.1/work/gross-1.0.1/config.log This file contains any messages produced by compilers while running configure, to aid debugging if configure makes a mistake. It was created by gross configure 1.0.1, which was generated by GNU Autoconf 2.61. Invocation command line was $ ./configure --prefix=/usr --host=i686-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --sysconfdir=/etc/mail/gross --disable-milter --disable-dnsbl --build=i686-pc-linux-gnu ## --------- ## ## Platform. ## ## --------- ## hostname = tisiphone uname -m = i686 uname -r = 2.6.28-rc7 uname -s = Linux uname -v = #2 SMP Sun Dec 7 01:19:41 CET 2008 /usr/bin/uname -p = Intel(R) Pentium(R) 4 CPU 2.80GHz /bin/uname -X = unknown /bin/arch = unknown /usr/bin/arch -k = unknown /usr/convex/getsysinfo = unknown /usr/bin/hostinfo = unknown /bin/machine = unknown /usr/bin/oslevel = unknown /bin/universe = unknown PATH: /usr/local/sbin PATH: /sbin PATH: /usr/sbin PATH: /usr/lib/portage/bin PATH: /usr/local/bin PATH: /bin PATH: /usr/bin PATH: /opt/bin PATH: /usr/i486-pc-linux-gnu/gcc-bin/4.1.2 PATH: /usr/i686-pc-linux-gnu/gcc-bin/4.3.2 ## ----------- ## ## Core tests. ## ## ----------- ## configure:1953: checking build system type configure:1971: result: i686-pc-linux-gnu configure:1993: checking host system type configure:2008: result: i686-pc-linux-gnu configure:2030: checking target system type configure:2045: result: i686-pc-linux-gnu configure:2088: checking for a BSD-compatible install configure:2144: result: /usr/bin/install -c configure:2155: checking whether build environment is sane configure:2198: result: yes configure:2226: checking for a thread-safe mkdir -p configure:2265: result: /bin/mkdir -p configure:2278: checking for gawk configure:2294: found /bin/gawk configure:2305: result: gawk configure:2316: checking whether make sets $(MAKE) configure:2337: result: yes configure:2540: checking for i686-pc-linux-gnu-cc configure:2570: result: no configure:2540: checking for i686-pc-linux-gnu-gcc configure:2556: found /usr/bin/i686-pc-linux-gnu-gcc configure:2567: result: i686-pc-linux-gnu-gcc configure:2647: checking for C compiler version configure:2654: i686-pc-linux-gnu-gcc --version >&5 i686-pc-linux-gnu-gcc (Gentoo 4.3.2 p1.2) 4.3.2 Copyright (C) 2008 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. configure:2657: $? = 0 configure:2664: i686-pc-linux-gnu-gcc -v >&5 Using built-in specs. Target: i686-pc-linux-gnu Configured with: /var/tmp/portage/sys-devel/gcc-4.3.2/work/gcc-4.3.2/configure --prefix=/usr --bindir=/usr/i686-pc-linux-gnu/gcc-bin/4.3.2 --includedir=/usr/lib/gcc/i686-pc-linux-gnu/4.3.2/include --datadir=/usr/share/gcc-data/i686-pc-linux-gnu/4.3.2 --mandir=/usr/share/gcc-data/i686-pc-linux-gnu/4.3.2/man --infodir=/usr/share/gcc-data/i686-pc-linux-gnu/4.3.2/info --with-gxx-include-dir=/usr/lib/gcc/i686-pc-linux-gnu/4.3.2/include/g++-v4 --host=i686-pc-linux-gnu --build=i686-pc-linux-gnu --disable-altivec --disable-fixed-point --enable-nls --without-included-gettext --with-system-zlib --disable-checking --disable-werror --enable-secureplt --disable-multilib --enable-libmudflap --disable-libssp --enable-libgomp --enable-cld --disable-libgcj --with-arch=i686 --enable-languages=c,c++,treelang,fortran --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --with-bugurl=http://bugs.gentoo.org/ --with-pkgversion='Gentoo 4.3.2 p1.2' Thread model: posix gcc version 4.3.2 (Gentoo 4.3.2 p1.2) configure:2667: $? = 0 configure:2674: i686-pc-linux-gnu-gcc -V >&5 i686-pc-linux-gnu-gcc: '-V' option must have argument configure:2677: $? = 1 configure:2700: checking for C compiler default output file name configure:2727: i686-pc-linux-gnu-gcc -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow -Wl,-O1 -Wl,--add-needed -Wl,--as-needed -Wl,--hash-style=both -Wl,--sort-common conftest.c >&5 configure:2730: $? = 0 configure:2768: result: a.out configure:2785: checking whether the C compiler works configure:2795: ./a.out configure:2798: $? = 0 configure:2815: result: yes configure:2822: checking whether we are cross compiling configure:2824: result: no configure:2827: checking for suffix of executables configure:2834: i686-pc-linux-gnu-gcc -o conftest -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow -Wl,-O1 -Wl,--add-needed -Wl,--as-needed -Wl,--hash-style=both -Wl,--sort-common conftest.c >&5 configure:2837: $? = 0 configure:2861: result: configure:2867: checking for suffix of object files configure:2893: i686-pc-linux-gnu-gcc -c -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow conftest.c >&5 configure:2896: $? = 0 configure:2919: result: o configure:2923: checking whether we are using the GNU C compiler configure:2952: i686-pc-linux-gnu-gcc -c -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow conftest.c >&5 configure:2958: $? = 0 configure:2975: result: yes configure:2980: checking whether i686-pc-linux-gnu-gcc accepts -g configure:3010: i686-pc-linux-gnu-gcc -c -g conftest.c >&5 configure:3016: $? = 0 configure:3115: result: yes configure:3132: checking for i686-pc-linux-gnu-gcc option to accept ISO C89 configure:3206: i686-pc-linux-gnu-gcc -c -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow conftest.c >&5 configure:3212: $? = 0 configure:3235: result: none needed configure:3264: checking for style of include used by make configure:3292: result: GNU configure:3317: checking dependency style of i686-pc-linux-gnu-gcc configure:3408: result: gcc3 configure:3436: checking for a BSD-compatible install configure:3492: result: /usr/bin/install -c configure:3579: checking for a sed that does not truncate output configure:3633: result: /bin/sed configure:3636: checking for grep that handles long lines and -e configure:3710: result: /bin/grep configure:3715: checking for egrep configure:3793: result: /bin/grep -E configure:3809: checking for ld used by i686-pc-linux-gnu-gcc configure:3876: result: /usr/i686-pc-linux-gnu/bin/ld configure:3885: checking if the linker (/usr/i686-pc-linux-gnu/bin/ld) is GNU ld configure:3900: result: yes configure:3905: checking for /usr/i686-pc-linux-gnu/bin/ld option to reload object files configure:3912: result: -r configure:3930: checking for BSD-compatible nm configure:3979: result: /usr/bin/nm -B configure:3983: checking whether ln -s works configure:3987: result: yes configure:3994: checking how to recognise dependent libraries configure:4170: result: pass_all configure:4404: checking how to run the C preprocessor configure:4444: i686-pc-linux-gnu-gcc -E conftest.c configure:4450: $? = 0 configure:4481: i686-pc-linux-gnu-gcc -E conftest.c conftest.c:10:28: error: ac_nonexistent.h: No such file or directory configure:4487: $? = 1 configure: failed program was: | /* confdefs.h. */ | #define PACKAGE_NAME "gross" | #define PACKAGE_TARNAME "gross" | #define PACKAGE_VERSION "1.0.1" | #define PACKAGE_STRING "gross 1.0.1" | #define PACKAGE_BUGREPORT "" | #define PACKAGE "gross" | #define VERSION "1.0.1" | /* end confdefs.h. */ | #include configure:4520: result: i686-pc-linux-gnu-gcc -E configure:4549: i686-pc-linux-gnu-gcc -E conftest.c configure:4555: $? = 0 configure:4586: i686-pc-linux-gnu-gcc -E conftest.c conftest.c:10:28: error: ac_nonexistent.h: No such file or directory configure:4592: $? = 1 configure: failed program was: | /* confdefs.h. */ | #define PACKAGE_NAME "gross" | #define PACKAGE_TARNAME "gross" | #define PACKAGE_VERSION "1.0.1" | #define PACKAGE_STRING "gross 1.0.1" | #define PACKAGE_BUGREPORT "" | #define PACKAGE "gross" | #define VERSION "1.0.1" | /* end confdefs.h. */ | #include configure:4630: checking for ANSI C header files configure:4660: i686-pc-linux-gnu-gcc -c -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow conftest.c >&5 configure:4666: $? = 0 configure:4765: i686-pc-linux-gnu-gcc -o conftest -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow -Wl,-O1 -Wl,--add-needed -Wl,--as-needed -Wl,--hash-style=both -Wl,--sort-common conftest.c >&5 configure:4768: $? = 0 configure:4774: ./conftest configure:4777: $? = 0 configure:4794: result: yes configure:4818: checking for sys/types.h configure:4839: i686-pc-linux-gnu-gcc -c -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow conftest.c >&5 configure:4845: $? = 0 configure:4861: result: yes configure:4818: checking for sys/stat.h configure:4839: i686-pc-linux-gnu-gcc -c -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow conftest.c >&5 configure:4845: $? = 0 configure:4861: result: yes configure:4818: checking for stdlib.h configure:4839: i686-pc-linux-gnu-gcc -c -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow conftest.c >&5 configure:4845: $? = 0 configure:4861: result: yes configure:4818: checking for string.h configure:4839: i686-pc-linux-gnu-gcc -c -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow conftest.c >&5 configure:4845: $? = 0 configure:4861: result: yes configure:4818: checking for memory.h configure:4839: i686-pc-linux-gnu-gcc -c -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow conftest.c >&5 configure:4845: $? = 0 configure:4861: result: yes configure:4818: checking for strings.h configure:4839: i686-pc-linux-gnu-gcc -c -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow conftest.c >&5 configure:4845: $? = 0 configure:4861: result: yes configure:4818: checking for inttypes.h configure:4839: i686-pc-linux-gnu-gcc -c -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow conftest.c >&5 configure:4845: $? = 0 configure:4861: result: yes configure:4818: checking for stdint.h configure:4839: i686-pc-linux-gnu-gcc -c -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow conftest.c >&5 configure:4845: $? = 0 configure:4861: result: yes configure:4818: checking for unistd.h configure:4839: i686-pc-linux-gnu-gcc -c -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow conftest.c >&5 configure:4845: $? = 0 configure:4861: result: yes configure:4888: checking dlfcn.h usability configure:4905: i686-pc-linux-gnu-gcc -c -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow conftest.c >&5 configure:4911: $? = 0 configure:4925: result: yes configure:4929: checking dlfcn.h presence configure:4944: i686-pc-linux-gnu-gcc -E conftest.c configure:4950: $? = 0 configure:4964: result: yes configure:4992: checking for dlfcn.h configure:5000: result: yes configure:5017: checking the maximum length of command line arguments configure:5126: result: 32768 configure:5137: checking command to parse /usr/bin/nm -B output from i686-pc-linux-gnu-gcc object configure:5242: i686-pc-linux-gnu-gcc -c -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow conftest.c >&5 configure:5245: $? = 0 configure:5249: /usr/bin/nm -B conftest.o \| sed -n -e 's/^.*[ ]\([ABCDGIRSTW][ABCDGIRSTW]*\)[ ][ ]*\([_A-Za-z][_A-Za-z0-9]*\)$/\1 \2 \2/p' \> conftest.nm configure:5252: $? = 0 configure:5304: i686-pc-linux-gnu-gcc -o conftest -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow -Wl,-O1 -Wl,--add-needed -Wl,--as-needed -Wl,--hash-style=both -Wl,--sort-common conftest.c conftstm.o >&5 configure:5307: $? = 0 configure:5345: result: ok configure:5349: checking for objdir configure:5364: result: .libs configure:5416: checking for i686-pc-linux-gnu-ar configure:5432: found /usr/bin/i686-pc-linux-gnu-ar configure:5443: result: i686-pc-linux-gnu-ar configure:5512: checking for i686-pc-linux-gnu-ranlib configure:5528: found /usr/bin/i686-pc-linux-gnu-ranlib configure:5539: result: i686-pc-linux-gnu-ranlib configure:5608: checking for i686-pc-linux-gnu-strip configure:5624: found /usr/bin/i686-pc-linux-gnu-strip configure:5635: result: i686-pc-linux-gnu-strip configure:5961: checking if i686-pc-linux-gnu-gcc supports -fno-rtti -fno-exceptions configure:5979: i686-pc-linux-gnu-gcc -c -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow -fno-rtti -fno-exceptions conftest.c >&5 cc1: warning: command line option "-fno-rtti" is valid for C++/ObjC++ but not for C configure:5983: $? = 0 configure:5996: result: no configure:6011: checking for i686-pc-linux-gnu-gcc option to produce PIC configure:6221: result: -fPIC configure:6229: checking if i686-pc-linux-gnu-gcc PIC flag -fPIC works configure:6247: i686-pc-linux-gnu-gcc -c -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow -fPIC -DPIC conftest.c >&5 configure:6251: $? = 0 configure:6264: result: yes configure:6292: checking if i686-pc-linux-gnu-gcc static flag -static works configure:6320: result: yes configure:6330: checking if i686-pc-linux-gnu-gcc supports -c -o file.o configure:6351: i686-pc-linux-gnu-gcc -c -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow -o out/conftest2.o conftest.c >&5 configure:6355: $? = 0 configure:6377: result: yes configure:6403: checking whether the i686-pc-linux-gnu-gcc linker (/usr/i686-pc-linux-gnu/bin/ld) supports shared libraries configure:7357: result: yes configure:7378: checking whether -lc should be explicitly linked in configure:7383: i686-pc-linux-gnu-gcc -c -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow conftest.c >&5 configure:7386: $? = 0 configure:7401: i686-pc-linux-gnu-gcc -shared conftest.o -v -Wl,-soname -Wl,conftest -o conftest 2\>\&1 \| grep -lc \>/dev/null 2\>\&1 configure:7404: $? = 0 configure:7416: result: no configure:7424: checking dynamic linker characteristics configure:8012: result: GNU/Linux ld.so configure:8021: checking how to hardcode library paths into programs configure:8046: result: immediate configure:8060: checking whether stripping libraries is possible configure:8065: result: yes configure:8866: checking if libtool supports shared libraries configure:8868: result: yes configure:8871: checking whether to build shared libraries configure:8892: result: yes configure:8895: checking whether to build static libraries configure:8899: result: yes configure:8991: creating libtool configure:9457: checking for library containing sem_init configure:9498: i686-pc-linux-gnu-gcc -o conftest -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow -I/usr/local/include -Wl,-O1 -Wl,--add-needed -Wl,--as-needed -Wl,--hash-style=both -Wl,--sort-common -L/usr/local/lib conftest.c >&5 /var/tmp/portage/mail-filter/gross-1.0.1/temp/ccMAP5M3.o: In function `main': conftest.c:(.text+0x12): undefined reference to `sem_init' collect2: ld returned 1 exit status configure:9504: $? = 1 configure: failed program was: | /* confdefs.h. */ | #define PACKAGE_NAME "gross" | #define PACKAGE_TARNAME "gross" | #define PACKAGE_VERSION "1.0.1" | #define PACKAGE_STRING "gross 1.0.1" | #define PACKAGE_BUGREPORT "" | #define PACKAGE "gross" | #define VERSION "1.0.1" | #define STDC_HEADERS 1 | #define HAVE_SYS_TYPES_H 1 | #define HAVE_SYS_STAT_H 1 | #define HAVE_STDLIB_H 1 | #define HAVE_STRING_H 1 | #define HAVE_MEMORY_H 1 | #define HAVE_STRINGS_H 1 | #define HAVE_INTTYPES_H 1 | #define HAVE_STDINT_H 1 | #define HAVE_UNISTD_H 1 | #define HAVE_DLFCN_H 1 | /* end confdefs.h. */ | | /* Override any GCC internal prototype to avoid an error. | Use char because int might match the return type of a GCC | builtin and then its argument prototype would still apply. */ | #ifdef __cplusplus | extern "C" | #endif | char sem_init (); | int | main () | { | return sem_init (); | ; | return 0; | } configure:9498: i686-pc-linux-gnu-gcc -o conftest -march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow -I/usr/local/include -Wl,-O1 -Wl,--add-needed -Wl,--as-needed -Wl,--hash-style=both -Wl,--sort-common -L/usr/local/lib conftest.c -lrt >&5 /var/tmp/portage/mail-filter/gross-1.0.1/temp/ccScp1Y9.o: In function `main': conftest.c:(.text+0x12): undefined reference to `sem_init' collect2: ld returned 1 exit status configure:9504: $? = 1 configure: failed program was: | /* confdefs.h. */ | #define PACKAGE_NAME "gross" | #define PACKAGE_TARNAME "gross" | #define PACKAGE_VERSION "1.0.1" | #define PACKAGE_STRING "gross 1.0.1" | #define PACKAGE_BUGREPORT "" | #define PACKAGE "gross" | #define VERSION "1.0.1" | #define STDC_HEADERS 1 | #define HAVE_SYS_TYPES_H 1 | #define HAVE_SYS_STAT_H 1 | #define HAVE_STDLIB_H 1 | #define HAVE_STRING_H 1 | #define HAVE_MEMORY_H 1 | #define HAVE_STRINGS_H 1 | #define HAVE_INTTYPES_H 1 | #define HAVE_STDINT_H 1 | #define HAVE_UNISTD_H 1 | #define HAVE_DLFCN_H 1 | /* end confdefs.h. */ | | /* Override any GCC internal prototype to avoid an error. | Use char because int might match the return type of a GCC | builtin and then its argument prototype would still apply. */ | #ifdef __cplusplus | extern "C" | #endif | char sem_init (); | int | main () | { | return sem_init (); | ; | return 0; | } configure:9532: result: no configure:9539: error: Can't compile without semaphores. ## ---------------- ## ## Cache variables. ## ## ---------------- ## ac_cv_build=i686-pc-linux-gnu ac_cv_c_compiler_gnu=yes ac_cv_env_CC_set= ac_cv_env_CC_value= ac_cv_env_CFLAGS_set=set ac_cv_env_CFLAGS_value='-march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow' ac_cv_env_CPPFLAGS_set= ac_cv_env_CPPFLAGS_value= ac_cv_env_CPP_set= ac_cv_env_CPP_value= ac_cv_env_LDFLAGS_set=set ac_cv_env_LDFLAGS_value='-Wl,-O1 -Wl,--add-needed -Wl,--as-needed -Wl,--hash-style=both -Wl,--sort-common' ac_cv_env_LIBS_set= ac_cv_env_LIBS_value= ac_cv_env_build_alias_set=set ac_cv_env_build_alias_value=i686-pc-linux-gnu ac_cv_env_host_alias_set=set ac_cv_env_host_alias_value=i686-pc-linux-gnu ac_cv_env_target_alias_set= ac_cv_env_target_alias_value= ac_cv_header_dlfcn_h=yes ac_cv_header_inttypes_h=yes ac_cv_header_memory_h=yes ac_cv_header_stdc=yes ac_cv_header_stdint_h=yes ac_cv_header_stdlib_h=yes ac_cv_header_string_h=yes ac_cv_header_strings_h=yes ac_cv_header_sys_stat_h=yes ac_cv_header_sys_types_h=yes ac_cv_header_unistd_h=yes ac_cv_host=i686-pc-linux-gnu ac_cv_objext=o ac_cv_path_EGREP='/bin/grep -E' ac_cv_path_GREP=/bin/grep ac_cv_path_install='/usr/bin/install -c' ac_cv_path_mkdir=/bin/mkdir ac_cv_prog_AR=i686-pc-linux-gnu-ar ac_cv_prog_AWK=gawk ac_cv_prog_CC=i686-pc-linux-gnu-gcc ac_cv_prog_CPP='i686-pc-linux-gnu-gcc -E' ac_cv_prog_RANLIB=i686-pc-linux-gnu-ranlib ac_cv_prog_STRIP=i686-pc-linux-gnu-strip ac_cv_prog_cc_c89= ac_cv_prog_cc_g=yes ac_cv_prog_make_make_set=yes ac_cv_search_sem_init=no ac_cv_target=i686-pc-linux-gnu am_cv_CC_dependencies_compiler_type=gcc3 lt_cv_deplibs_check_method=pass_all lt_cv_file_magic_cmd='$MAGIC_CMD' lt_cv_file_magic_test_file= lt_cv_ld_reload_flag=-r lt_cv_objdir=.libs lt_cv_path_LD=/usr/i686-pc-linux-gnu/bin/ld lt_cv_path_NM='/usr/bin/nm -B' lt_cv_path_SED=/bin/sed lt_cv_prog_compiler_c_o=yes lt_cv_prog_compiler_rtti_exceptions=no lt_cv_prog_gnu_ld=yes lt_cv_sys_global_symbol_pipe='sed -n -e '\''s/^.*[ ]\([ABCDGIRSTW][ABCDGIRSTW]*\)[ ][ ]*\([_A-Za-z][_A-Za-z0-9]*\)$/\1 \2 \2/p'\''' lt_cv_sys_global_symbol_to_c_name_address='sed -n -e '\''s/^: \([^ ]*\) $/ {\"\1\", (lt_ptr) 0},/p'\'' -e '\''s/^[BCDEGRST] \([^ ]*\) \([^ ]*\)$/ {"\2", (lt_ptr) \&\2},/p'\''' lt_cv_sys_global_symbol_to_cdecl='sed -n -e '\''s/^. .* \(.*\)$/extern int \1;/p'\''' lt_cv_sys_max_cmd_len=32768 lt_lt_cv_prog_compiler_c_o='"yes"' lt_lt_cv_sys_global_symbol_pipe='"sed -n -e '\''s/^.*[ ]\\([ABCDGIRSTW][ABCDGIRSTW]*\\)[ ][ ]*\\([_A-Za-z][_A-Za-z0-9]*\\)\$/\\1 \\2 \\2/p'\''"' lt_lt_cv_sys_global_symbol_to_c_name_address='"sed -n -e '\''s/^: \\([^ ]*\\) \$/ {\\\"\\1\\\", (lt_ptr) 0},/p'\'' -e '\''s/^[BCDEGRST] \\([^ ]*\\) \\([^ ]*\\)\$/ {\"\\2\", (lt_ptr) \\&\\2},/p'\''"' lt_lt_cv_sys_global_symbol_to_cdecl='"sed -n -e '\''s/^. .* \\(.*\\)\$/extern int \\1;/p'\''"' ## ----------------- ## ## Output variables. ## ## ----------------- ## ACLOCAL='${SHELL} /var/tmp/portage/mail-filter/gross-1.0.1/work/gross-1.0.1/missing --run aclocal-1.10' AMDEPBACKSLASH='\' AMDEP_FALSE='#' AMDEP_TRUE='' AMTAR='${SHELL} /var/tmp/portage/mail-filter/gross-1.0.1/work/gross-1.0.1/missing --run tar' AR='i686-pc-linux-gnu-ar' AUTOCONF='${SHELL} /var/tmp/portage/mail-filter/gross-1.0.1/work/gross-1.0.1/missing --run autoconf' AUTOHEADER='${SHELL} /var/tmp/portage/mail-filter/gross-1.0.1/work/gross-1.0.1/missing --run autoheader' AUTOMAKE='${SHELL} /var/tmp/portage/mail-filter/gross-1.0.1/work/gross-1.0.1/missing --run automake-1.10' AWK='gawk' CC='i686-pc-linux-gnu-gcc' CCDEPMODE='depmode=gcc3' CFLAGS='-march=native -O2 -pipe -fweb -frename-registers -ftree-vectorize -freorder-blocks-and-partition -fgcse-sm -fgcse-las -maccumulate-outgoing-args -ftree-loop-optimize -funswitch-loops -ftracer -fprefetch-loop-arrays -fno-ident -fno-strict-overflow' CPP='i686-pc-linux-gnu-gcc -E' CPPFLAGS=' -I/usr/local/include' CYGPATH_W='echo' DEFS='' DEPDIR='.deps' DNSBLSOURCES='' ECHO='echo' ECHO_C='' ECHO_N='-n' ECHO_T='' EGREP='/bin/grep -E' EXEEXT='' GREP='/bin/grep' HAVE_SED_FALSE='#' HAVE_SED_TRUE='' INSTALL_DATA='${INSTALL} -m 644' INSTALL_PROGRAM='${INSTALL}' INSTALL_SCRIPT='${INSTALL}' INSTALL_STRIP_PROGRAM='$(install_sh) -c -s' LDFLAGS='-Wl,-O1 -Wl,--add-needed -Wl,--as-needed -Wl,--hash-style=both -Wl,--sort-common -L/usr/local/lib' LIBOBJS='' LIBS='' LIBTOOL='$(SHELL) $(top_builddir)/libtool' LN_S='ln -s' LTLIBOBJS='' MAKEINFO='${SHELL} /var/tmp/portage/mail-filter/gross-1.0.1/work/gross-1.0.1/missing --run makeinfo' MILTERSOURCES='' OBJEXT='o' PACKAGE='gross' PACKAGE_BUGREPORT='' PACKAGE_NAME='gross' PACKAGE_STRING='gross 1.0.1' PACKAGE_TARNAME='gross' PACKAGE_VERSION='1.0.1' PATH_SEPARATOR=':' RANLIB='i686-pc-linux-gnu-ranlib' REENTRANT_FLAG='-D_REENTRANT' SED='/bin/sed' SET_MAKE='' SHELL='/bin/sh' STATIC_GLIBC_FLAG='' STRIP='i686-pc-linux-gnu-strip' VERSION='1.0.1' ac_ct_CC='' am__fastdepCC_FALSE='#' am__fastdepCC_TRUE='' am__include='include' am__isrc='' am__leading_dot='.' am__quote='' am__tar='${AMTAR} chof - "$$tardir"' am__untar='${AMTAR} xf -' bindir='${exec_prefix}/bin' build='i686-pc-linux-gnu' build_alias='i686-pc-linux-gnu' build_cpu='i686' build_os='linux-gnu' build_vendor='pc' datadir='/usr/share' datarootdir='${prefix}/share' docdir='${datarootdir}/doc/${PACKAGE_TARNAME}' dvidir='${docdir}' exec_prefix='NONE' host='i686-pc-linux-gnu' host_alias='i686-pc-linux-gnu' host_cpu='i686' host_os='linux-gnu' host_vendor='pc' htmldir='${docdir}' includedir='${prefix}/include' infodir='/usr/share/info' install_sh='$(SHELL) /var/tmp/portage/mail-filter/gross-1.0.1/work/gross-1.0.1/install-sh' libdir='${exec_prefix}/lib' libexecdir='${exec_prefix}/libexec' localedir='${datarootdir}/locale' localstatedir='/var/lib' mandir='/usr/share/man' mkdir_p='/bin/mkdir -p' oldincludedir='/usr/include' pdfdir='${docdir}' prefix='/usr' program_transform_name='s,x,x,' psdir='${docdir}' sbindir='${exec_prefix}/sbin' sharedstatedir='${prefix}/com' sysconfdir='/etc/mail/gross' target='i686-pc-linux-gnu' target_alias='' target_cpu='i686' target_os='linux-gnu' target_vendor='pc' ## ----------- ## ## confdefs.h. ## ## ----------- ## #define PACKAGE_NAME "gross" #define PACKAGE_TARNAME "gross" #define PACKAGE_VERSION "1.0.1" #define PACKAGE_STRING "gross 1.0.1" #define PACKAGE_BUGREPORT "" #define PACKAGE "gross" #define VERSION "1.0.1" #define STDC_HEADERS 1 #define HAVE_SYS_TYPES_H 1 #define HAVE_SYS_STAT_H 1 #define HAVE_STDLIB_H 1 #define HAVE_STRING_H 1 #define HAVE_MEMORY_H 1 #define HAVE_STRINGS_H 1 #define HAVE_INTTYPES_H 1 #define HAVE_STDINT_H 1 #define HAVE_UNISTD_H 1 #define HAVE_DLFCN_H 1 configure: exit 1 tisiphone ~ # ----- You know what? I see it now. It must be that here: --- LDFLAGS='-Wl,-O1 -Wl,--add-needed -Wl,--as-needed -Wl,--hash-style=both -Wl,--sort-common -L/usr/local/lib' --- Let me check to filter "-Wl,--add-needed" and "-Wl,--as-needed" out of the LDFLAGS and look if that works. ... No. That did not work. Let me try to just remove the "-Wl,--as-needed". ... YES! That works! I can compile GROSS. Just had to inherit "flag-o-matic" and add "filter-ldflags -Wl,--as-needed" before the configure command into my ebuild and things work now. Sorry for the noise. If some one is interested in a Gentoo ebuild then let me know. // Steve -------- Original-Nachricht -------- > Datum: Tue, 09 Dec 2008 13:33:39 +0200 > Von: Eino Tuominen > An: Steve > CC: gross at lists.utu.fi > Betreff: Re: [Gross] Can\'t run configure on system with glibc-2.7 > Steve wrote: > > > > checking for library containing sem_init... no > > configure: error: Can't compile without semaphores. > > tisiphone ~ # > > ---- > > > > But everyting is there: > > ---- > > tisiphone ~ # grep -R "sem_init" /usr/include/ > > /usr/include/semaphore.h:extern int sem_init (sem_t *__sem, int > __pshared, unsigned int __value) > > tisiphone ~ # ls -lah /usr/include/semaphore.h > > -rw-r--r-- 1 root root 2.5K Dec 7 13:20 /usr/include/semaphore.h > > tisiphone ~ # > > ---- > > Hi, > > Check config.log first. There you should find the test program configure > is trying to compile. It may be that configure is unable to find the > library function for some reason. > > -- > Eino Tuominen -- Psssst! Schon vom neuen GMX MultiMessenger geh?rt? Der kann`s mit allen: http://www.gmx.net/de/go/multimessenger From jesse.thompson at doit.wisc.edu Fri Dec 12 21:52:16 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Fri, 12 Dec 2008 13:52:16 -0600 Subject: [Gross] which dnsbls? Message-ID: <4942C0F0.5090102@doit.wisc.edu> Which dnsbls are people using with gross? we're using: bl.spamcop.net cbl.abuseat.org dnsbl.sorbs.net dnsbl-1.uceprotect.net dnsbl-2.uceprotect.net dnsbl-3.uceprotect.net dnsbl.njabl.org and the sophos blocker However, I'm thinking of disabling njabl, since we've had three cases of semi-legitimate mass mailers being listed on njabl. Since we also do connection limiting, the greylisting was causing these large mailers to reconnect enough to run into our connection limiting. Does anyone have any additional thoughts on njabl? Jesse -- Jesse Thompson Division of Information Technology, University of Wisconsin-Madison Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From jpiszcz at lucidpixels.com Fri Dec 12 22:30:38 2008 From: jpiszcz at lucidpixels.com (Justin Piszcz) Date: Fri, 12 Dec 2008 15:30:38 -0500 (EST) Subject: [Gross] which dnsbls? In-Reply-To: <4942C0F0.5090102@doit.wisc.edu> References: <4942C0F0.5090102@doit.wisc.edu> Message-ID: On Fri, 12 Dec 2008, Jesse Thompson wrote: > Which dnsbls are people using with gross? > > we're using: > bl.spamcop.net > cbl.abuseat.org > dnsbl.sorbs.net > dnsbl-1.uceprotect.net > dnsbl-2.uceprotect.net > dnsbl-3.uceprotect.net > dnsbl.njabl.org > and the sophos blocker > > However, I'm thinking of disabling njabl, since we've had three cases of > semi-legitimate mass mailers being listed on njabl. Since we also do > connection limiting, the greylisting was causing these large mailers to > reconnect enough to run into our connection limiting. Does anyone have any > additional thoughts on njabl? Hi Jesse, -> bhnc.njabl.org This one is a bit safer, I have also seen false-positives with the one you mention. In addition, dnsbl-1.uceprotect.net has also blocked IBM once, but you just need to set the tolerances higher if using dnsbl-X.uce*. cbl.abuseat.org also false positives occasionally bl.spamcop.net & dnsbl.sorbs.net very accurate and result in few false positives (in my experience) As far as the sophos blocker, I have not tried/used it. Justin. From jesse.thompson at doit.wisc.edu Fri Dec 12 23:35:17 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Fri, 12 Dec 2008 15:35:17 -0600 Subject: [Gross] which dnsbls? In-Reply-To: References: <4942C0F0.5090102@doit.wisc.edu> Message-ID: <4942D915.7070001@doit.wisc.edu> Justin Piszcz wrote: > > > On Fri, 12 Dec 2008, Jesse Thompson wrote: > >> Which dnsbls are people using with gross? >> >> we're using: >> bl.spamcop.net >> cbl.abuseat.org >> dnsbl.sorbs.net >> dnsbl-1.uceprotect.net >> dnsbl-2.uceprotect.net >> dnsbl-3.uceprotect.net >> dnsbl.njabl.org >> and the sophos blocker >> >> However, I'm thinking of disabling njabl, since we've had three cases >> of semi-legitimate mass mailers being listed on njabl. Since we also >> do connection limiting, the greylisting was causing these large >> mailers to reconnect enough to run into our connection limiting. Does >> anyone have any additional thoughts on njabl? > > Hi Jesse, > > -> bhnc.njabl.org > > This one is a bit safer, I have also seen false-positives with the one > you mention. Cool. I'll try it. > In addition, dnsbl-1.uceprotect.net has also blocked IBM once, but you just > need to set the tolerances higher if using dnsbl-X.uce*. Yeah, I knew that was an aggressive one. But, we haven't had any complaints about it yet. > cbl.abuseat.org also false positives occasionally > > bl.spamcop.net & dnsbl.sorbs.net very accurate and result in few false > positives (in my experience) > > As far as the sophos blocker, I have not tried/used it. You have to be running Sophos PureMessage (or maybe a related product) to use it. Jesse -- Jesse Thompson Division of Information Technology, University of Wisconsin-Madison Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From Jarno.Huuskonen at uku.fi Mon Dec 15 09:50:39 2008 From: Jarno.Huuskonen at uku.fi (Jarno Huuskonen) Date: Mon, 15 Dec 2008 09:50:39 +0200 Subject: [Gross] which dnsbls? In-Reply-To: <4942C0F0.5090102@doit.wisc.edu> References: <4942C0F0.5090102@doit.wisc.edu> Message-ID: <20081215075038.GB8103@lossi76.uku.fi> On Fri, Dec 12, Jesse Thompson wrote: > Which dnsbls are people using with gross? > > we're using: > bl.spamcop.net > cbl.abuseat.org > dnsbl.sorbs.net > dnsbl-1.uceprotect.net > dnsbl-2.uceprotect.net > dnsbl-3.uceprotect.net > dnsbl.njabl.org > and the sophos blocker bl.spamcop.net b.barracudacentral.org dnsbl-1.uceprotect.net dnsbl-2.uceprotect.net dnsbl-3.uceprotect.net dnsbl.sorbs.net psbl.surriel.com combined.njabl.org (And we block mail with zen.spamhaus.org and trend micro rbl+). barracudacentral and uceprotect.net seem most effective. -Jarno -- Jarno Huuskonen University of Kuopio - Computer Center From imriz at bsd.org.il Tue Dec 16 13:26:26 2008 From: imriz at bsd.org.il (Imri Zvik) Date: Tue, 16 Dec 2008 13:26:26 +0200 Subject: [Gross] false positives Message-ID: <200812161326.26177.imriz@bsd.org.il> Hi, I have two grossd server with the following configuration: number_buffers=32 rotate_interval=10800 After running for a couple of days, the rate of false positives increases, and now most of the attempts results false positives: [root at greylist2 ~]# /usr/local/bin/gclient sjsms somethingelse at abc.com greylist at me.com 137.0.0.2 j823wd23ed 2 10.220.17.1 1111 got: M $Y M $Y got: M $Y M $Y [root at greylist2 ~]# /usr/local/bin/gclient sjsms somethingelse at monty.com greylist at somewhere.com 111.111.111.111 f34f3ggsdgf 2 10.220.17.1 1111 got: M $Y M $Y got: M $Y M $Y grossd summary: Dec 16 13:23:33 greylist2 grossd: grossd status summary (begin, end, trust, match, greylist, block): 1229426312, 1229426613, 0, 10452, 1060, 0 From jesse.thompson at doit.wisc.edu Tue Dec 16 18:00:53 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Tue, 16 Dec 2008 10:00:53 -0600 Subject: [Gross] false positives In-Reply-To: <200812161326.26177.imriz@bsd.org.il> References: <200812161326.26177.imriz@bsd.org.il> Message-ID: <4947D0B5.8000102@doit.wisc.edu> You probably need to raise filter_bits setting; especially since you are using such a large rotate_interval. Or, if you use the block_threshold functionality, most messages will be blocked and not added to the filters, which reduces the chance of false positives. I ran into a similar problem a while ago. I had raised the filter_bits setting as high as 25 to get rid of the false positives. Now that we use the block feature, I have lowered it to 22. But again, this all depends on your rotate_interval setting as well as the load on your servers. Our current settings: num_buffers: 12 rotate_interval = 3600 filter_bits = 22 Jesse Imri Zvik wrote: > Hi, > > I have two grossd server with the following configuration: > > number_buffers=32 > rotate_interval=10800 > > > > After running for a couple of days, the rate of false positives increases, and > now most of the attempts results false positives: > > > [root at greylist2 ~]# /usr/local/bin/gclient sjsms somethingelse at abc.com > greylist at me.com 137.0.0.2 j823wd23ed 2 10.220.17.1 1111 > got: M $Y > M $Y > got: M $Y > M $Y > [root at greylist2 ~]# /usr/local/bin/gclient sjsms somethingelse at monty.com > greylist at somewhere.com 111.111.111.111 f34f3ggsdgf 2 10.220.17.1 1111 > got: M $Y > M $Y > got: M $Y > M $Y > > > > grossd summary: > > Dec 16 13:23:33 greylist2 grossd: grossd status summary (begin, end, trust, > match, greylist, block): 1229426312, 1229426613, 0, 10452, 1060, 0 > > > _______________________________________________ > Gross mailing list > Gross at lists.utu.fi > https://lists.utu.fi/mailman/listinfo/gross -- Jesse Thompson Division of Information Technology, University of Wisconsin-Madison Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Wed Dec 17 14:22:51 2008 From: eino at utu.fi (Eino Tuominen) Date: Wed, 17 Dec 2008 14:22:51 +0200 Subject: [Gross] false positives In-Reply-To: <4947D0B5.8000102@doit.wisc.edu> References: <200812161326.26177.imriz@bsd.org.il> <4947D0B5.8000102@doit.wisc.edu> Message-ID: <4948EF1B.1060700@utu.fi> Jesse Thompson wrote: > You probably need to raise filter_bits setting; especially since you are > using such a large rotate_interval. > > Or, if you use the block_threshold functionality, most messages will be > blocked and not added to the filters, which reduces the chance of false > positives. Indeed, that's seems to be the best way. Just for the record, we are now talking false matches, or collisions in the bloom filter, not false positives. -- Eino Tuominen From imriz at bsd.org.il Wed Dec 17 15:37:00 2008 From: imriz at bsd.org.il (Imri Zvik) Date: Wed, 17 Dec 2008 15:37:00 +0200 Subject: [Gross] false positives In-Reply-To: <4948EF1B.1060700@utu.fi> References: <200812161326.26177.imriz@bsd.org.il> <4947D0B5.8000102@doit.wisc.edu> <4948EF1B.1060700@utu.fi> Message-ID: <200812171537.00444.imriz@bsd.org.il> On Wednesday 17 December 2008 14:22:51 Eino Tuominen wrote: > Jesse Thompson wrote: > > You probably need to raise filter_bits setting; especially since you are > > using such a large rotate_interval. > > > > Or, if you use the block_threshold functionality, most messages will be > > blocked and not added to the filters, which reduces the chance of false > > positives. > > Indeed, that's seems to be the best way. > > Just for the record, we are now talking false matches, or collisions in > the bloom filter, not false positives. I forgot to CC the list on my reply to Jesse: If I raise the filter bit, grossd crashes with segmentation fault. Linux greylist2 2.6.9-55.ELsmp #1 SMP Fri Apr 20 16:36:54 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux From jesse.thompson at doit.wisc.edu Wed Dec 17 19:10:33 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Wed, 17 Dec 2008 11:10:33 -0600 Subject: [Gross] false positives In-Reply-To: <200812171537.00444.imriz@bsd.org.il> References: <200812161326.26177.imriz@bsd.org.il> <4947D0B5.8000102@doit.wisc.edu> <4948EF1B.1060700@utu.fi> <200812171537.00444.imriz@bsd.org.il> Message-ID: <49493289.1080708@doit.wisc.edu> Imri Zvik wrote: > On Wednesday 17 December 2008 14:22:51 Eino Tuominen wrote: >> Jesse Thompson wrote: >>> You probably need to raise filter_bits setting; especially since you are >>> using such a large rotate_interval. >>> >>> Or, if you use the block_threshold functionality, most messages will be >>> blocked and not added to the filters, which reduces the chance of false >>> positives. >> Indeed, that's seems to be the best way. >> >> Just for the record, we are now talking false matches, or collisions in >> the bloom filter, not false positives. > > I forgot to CC the list on my reply to Jesse: > > If I raise the filter bit, grossd crashes with segmentation fault. > > Linux greylist2 2.6.9-55.ELsmp #1 SMP Fri Apr 20 16:36:54 EDT 2007 x86_64 > x86_64 x86_64 GNU/Linux During the initial filter replication? We saw a similar problem, but it was triggered by Solaris 10 patching. We were never able to figure out the root cause. Lowering the filter_bits allowed the filters to sync before it crashed. Since you can't raise the filter_bits, can you lower the rotate_interval? That should lower the chance of false matches. Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From eino at utu.fi Wed Dec 17 19:16:07 2008 From: eino at utu.fi (Eino Tuominen) Date: Wed, 17 Dec 2008 19:16:07 +0200 Subject: [Gross] false positives In-Reply-To: <200812171537.00444.imriz@bsd.org.il> References: <200812161326.26177.imriz@bsd.org.il> <4947D0B5.8000102@doit.wisc.edu> <4948EF1B.1060700@utu.fi> <200812171537.00444.imriz@bsd.org.il> Message-ID: <494933D7.3010609@utu.fi> Imri Zvik wrote: > > I forgot to CC the list on my reply to Jesse: > > If I raise the filter bit, grossd crashes with segmentation fault. Please, show your configuration file that triggered the crash. At least filter_bits, number_buffers and rotate_interval. -- Eino Tuominen From imriz at bsd.org.il Thu Dec 18 09:29:40 2008 From: imriz at bsd.org.il (Imri Zvik) Date: Thu, 18 Dec 2008 09:29:40 +0200 Subject: [Gross] false positives In-Reply-To: <49493289.1080708@doit.wisc.edu> References: <200812161326.26177.imriz@bsd.org.il> <200812171537.00444.imriz@bsd.org.il> <49493289.1080708@doit.wisc.edu> Message-ID: <200812180929.40720.imriz@bsd.org.il> On Wednesday 17 December 2008 19:10:33 Jesse Thompson wrote: > Imri Zvik wrote: > > On Wednesday 17 December 2008 14:22:51 Eino Tuominen wrote: > >> Jesse Thompson wrote: > >>> You probably need to raise filter_bits setting; especially since you > >>> are using such a large rotate_interval. > >>> > >>> Or, if you use the block_threshold functionality, most messages will be > >>> blocked and not added to the filters, which reduces the chance of false > >>> positives. > >> > >> Indeed, that's seems to be the best way. > >> > >> Just for the record, we are now talking false matches, or collisions in > >> the bloom filter, not false positives. > > > > I forgot to CC the list on my reply to Jesse: > > > > If I raise the filter bit, grossd crashes with segmentation fault. > > > > Linux greylist2 2.6.9-55.ELsmp #1 SMP Fri Apr 20 16:36:54 EDT 2007 x86_64 > > x86_64 x86_64 GNU/Linux > > During the initial filter replication? We saw a similar problem, but it > was triggered by Solaris 10 patching. We were never able to figure out > the root cause. Lowering the filter_bits allowed the filters to sync > before it crashed. Yes, it crashed a few seconds after startup, while replicating with the other perr. > > Since you can't raise the filter_bits, can you lower the > rotate_interval? That should lower the chance of false matches. No, as this would require remote MTAs to go through the greylisting process more often. We would like to keep a match for at least a week ~ > > Jesse Thank you both for your efforts. I must say that compared to other greylisting implementations, yours just fly! From imriz at bsd.org.il Thu Dec 18 09:25:33 2008 From: imriz at bsd.org.il (Imri Zvik) Date: Thu, 18 Dec 2008 09:25:33 +0200 Subject: [Gross] false positives In-Reply-To: <494933D7.3010609@utu.fi> References: <200812161326.26177.imriz@bsd.org.il> <200812171537.00444.imriz@bsd.org.il> <494933D7.3010609@utu.fi> Message-ID: <200812180925.33031.imriz@bsd.org.il> On Wednesday 17 December 2008 19:16:07 Eino Tuominen wrote: > Imri Zvik wrote: > > I forgot to CC the list on my reply to Jesse: > > > > If I raise the filter bit, grossd crashes with segmentation fault. > > Please, show your configuration file that triggered the crash. At least > filter_bits, number_buffers and rotate_interval. filter_bits = 26 number_buffers=32 rotate_interval=10800 If I raise the filter_bits anywhere above 26, it would crash after a few seconds (while replicating/syncing with the other peer). From eino at utu.fi Thu Dec 18 10:13:31 2008 From: eino at utu.fi (Eino Tuominen) Date: Thu, 18 Dec 2008 10:13:31 +0200 Subject: [Gross] false positives In-Reply-To: <200812180925.33031.imriz@bsd.org.il> References: <200812161326.26177.imriz@bsd.org.il> <200812171537.00444.imriz@bsd.org.il> <494933D7.3010609@utu.fi> <200812180925.33031.imriz@bsd.org.il> Message-ID: <494A062B.6060606@utu.fi> Imri Zvik wrote: > On Wednesday 17 December 2008 19:16:07 Eino Tuominen wrote: >> Imri Zvik wrote: >>> I forgot to CC the list on my reply to Jesse: >>> >>> If I raise the filter bit, grossd crashes with segmentation fault. >> Please, show your configuration file that triggered the crash. At least >> filter_bits, number_buffers and rotate_interval. > > > filter_bits = 26 > number_buffers=32 > rotate_interval=10800 > > If I raise the filter_bits anywhere above 26, it would crash after a few > seconds (while replicating/syncing with the other peer). With your settings you end up using over 2 gigabytes of memory - if you increase filter_bits even by one it will use more than 4 GB. Which is more than a 32 bit virtual memory can map. I suggest you start using block_treshold feature. If that's really not an option, try something like this: filter_bits = 28 number_buffers = 8 rotate_interval = 43200 Also, your expire time is a bit long, 24 hours should be enough always. filter_bits = 28 number_buffers = 9 rotate_interval = 10800 You can calculate the needed memory in bytes with this formula: 2^filter_bits * ( number_buffers + 1 ) Filter expiration time is (in seconds) number_buffers * rotate interval Hope this helps, -- Eino Tuominen From eino at utu.fi Thu Dec 18 10:51:18 2008 From: eino at utu.fi (Eino Tuominen) Date: Thu, 18 Dec 2008 10:51:18 +0200 Subject: [Gross] false positives In-Reply-To: <200812181024.11902.imriz@bsd.org.il> References: <200812161326.26177.imriz@bsd.org.il> <494A062B.6060606@utu.fi> <200812181022.04510.imriz@bsd.org.il> <200812181024.11902.imriz@bsd.org.il> Message-ID: <494A0F06.7040300@utu.fi> Imri Zvik wrote: > > By the way, grossd is taking much less than 2 GB: > > nobody 22963 0.4 6.7 448440 273280 ? Sl Dec07 > 65:07 /usr/local/sbin/grossd Ah, my bad. Note to self: a byte is eight bits... ;-) Well, I managed to duplicate the problem, it seems that there's something wrong in the bloom filter setup in the code. I'll have to examine it more... So, for now you are left with the workarounds of playing with number_buffers and rotate_interval. -- Eino Tuominen From imriz at bsd.org.il Thu Dec 18 13:37:22 2008 From: imriz at bsd.org.il (Imri Zvik) Date: Thu, 18 Dec 2008 13:37:22 +0200 Subject: [Gross] false positives In-Reply-To: <494A0F06.7040300@utu.fi> References: <200812161326.26177.imriz@bsd.org.il> <200812181024.11902.imriz@bsd.org.il> <494A0F06.7040300@utu.fi> Message-ID: <200812181337.22452.imriz@bsd.org.il> On Thursday 18 December 2008 10:51:18 Eino Tuominen wrote: > Imri Zvik wrote: > > By the way, grossd is taking much less than 2 GB: > > > > nobody 22963 0.4 6.7 448440 273280 ? Sl Dec07 > > 65:07 /usr/local/sbin/grossd > > Ah, my bad. Note to self: a byte is eight bits... ;-) > > Well, I managed to duplicate the problem, it seems that there's > something wrong in the bloom filter setup in the code. I'll have to > examine it more... > > So, for now you are left with the workarounds of playing with > number_buffers and rotate_interval. Well, a problem reproduced is a problem half solved ;) (Should I reopen Issue 78?) Thanks for the help! PS Are you announcing new versions in this list? From jesse.thompson at doit.wisc.edu Thu Dec 18 16:11:12 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Thu, 18 Dec 2008 08:11:12 -0600 Subject: [Gross] false positives In-Reply-To: <200812181337.22452.imriz@bsd.org.il> References: <200812161326.26177.imriz@bsd.org.il> <200812181024.11902.imriz@bsd.org.il> <494A0F06.7040300@utu.fi> <200812181337.22452.imriz@bsd.org.il> Message-ID: <494A5A00.1030209@doit.wisc.edu> Imri Zvik wrote: > On Thursday 18 December 2008 10:51:18 Eino Tuominen wrote: >> Imri Zvik wrote: >>> By the way, grossd is taking much less than 2 GB: >>> >>> nobody 22963 0.4 6.7 448440 273280 ? Sl Dec07 >>> 65:07 /usr/local/sbin/grossd >> Ah, my bad. Note to self: a byte is eight bits... ;-) >> >> Well, I managed to duplicate the problem, it seems that there's >> something wrong in the bloom filter setup in the code. I'll have to >> examine it more... >> >> So, for now you are left with the workarounds of playing with >> number_buffers and rotate_interval. > > > Well, a problem reproduced is a problem half solved ;) > (Should I reopen Issue 78?) > > Thanks for the help! Thanks for persisting. That bug (assuming it's the same one) was a pain in the ass for us. I hope Eino can resolve it. Jesse > > > PS > Are you announcing new versions in this list? > > > _______________________________________________ > Gross mailing list > Gross at lists.utu.fi > https://lists.utu.fi/mailman/listinfo/gross -- Jesse Thompson Division of Information Technology, University of Wisconsin-Madison Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From jesse.thompson at doit.wisc.edu Thu Dec 18 16:18:54 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Thu, 18 Dec 2008 08:18:54 -0600 Subject: [Gross] false positives In-Reply-To: <200812180929.40720.imriz@bsd.org.il> References: <200812161326.26177.imriz@bsd.org.il> <200812171537.00444.imriz@bsd.org.il> <49493289.1080708@doit.wisc.edu> <200812180929.40720.imriz@bsd.org.il> Message-ID: <494A5BCE.80304@doit.wisc.edu> Imri Zvik wrote: > We would like to keep a match for at least a week ~ I'm curious... what's your reasoning? Jesse -- Jesse Thompson Division of Information Technology, University of Wisconsin-Madison Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From jesse.thompson at doit.wisc.edu Thu Dec 18 16:38:32 2008 From: jesse.thompson at doit.wisc.edu (Jesse Thompson) Date: Thu, 18 Dec 2008 08:38:32 -0600 Subject: [Gross] false positives In-Reply-To: <200812181624.33796.imriz@inter.net.il> References: <200812161326.26177.imriz@bsd.org.il> <200812180929.40720.imriz@bsd.org.il> <494A5BCE.80304@doit.wisc.edu> <200812181624.33796.imriz@inter.net.il> Message-ID: <494A6068.7080006@doit.wisc.edu> Imri Zvik wrote: > On Thursday 18 December 2008 16:18:54 Jesse Thompson wrote: >> Imri Zvik wrote: >>> We would like to keep a match for at least a week ~ >> I'm curious... what's your reasoning? > Some users complained about the delay - if the triplet is kept for more time, > the less it would be noticeable. I'm an ISP, so I need to actually listen to > what my users has to say :) It can't be as bad as an EDU, when most people know how to complain directly to the CIO. :-) You must be doing traditional greylisting, instead of using DNSBLs to determine which IPs to greylist. Jesse -- Jesse Thompson Division of Information Technology, University of Wisconsin-Madison Email/IM: jesse.thompson at doit.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3340 bytes Desc: S/MIME Cryptographic Signature URL: From hege at hege.li Thu Dec 18 17:16:50 2008 From: hege at hege.li (Henrik Krohns) Date: Thu, 18 Dec 2008 17:16:50 +0200 Subject: [Gross] false positives In-Reply-To: <494A6068.7080006@doit.wisc.edu> References: <200812161326.26177.imriz@bsd.org.il> <200812180929.40720.imriz@bsd.org.il> <494A5BCE.80304@doit.wisc.edu> <200812181624.33796.imriz@inter.net.il> <494A6068.7080006@doit.wisc.edu> Message-ID: <20081218151650.GA5810@posti.hege.li> On Thu, Dec 18, 2008 at 08:38:32AM -0600, Jesse Thompson wrote: > Imri Zvik wrote: >> On Thursday 18 December 2008 16:18:54 Jesse Thompson wrote: >>> Imri Zvik wrote: >>>> We would like to keep a match for at least a week ~ >>> I'm curious... what's your reasoning? >> Some users complained about the delay - if the triplet is kept for more >> time, the less it would be noticeable. I'm an ISP, so I need to >> actually listen to what my users has to say :) > > It can't be as bad as an EDU, when most people know how to complain > directly to the CIO. :-) > > You must be doing traditional greylisting, instead of using DNSBLs to > determine which IPs to greylist. It seems pretty clear to me that grossd is missing auto-whitelist feature. What's the point greylisting legimate retrying hosts, that might only send once a day or so? From imriz at inter.net.il Thu Dec 18 16:24:33 2008 From: imriz at inter.net.il (Imri Zvik) Date: Thu, 18 Dec 2008 16:24:33 +0200 Subject: [Gross] false positives In-Reply-To: <494A5BCE.80304@doit.wisc.edu> References: <200812161326.26177.imriz@bsd.org.il> <200812180929.40720.imriz@bsd.org.il> <494A5BCE.80304@doit.wisc.edu> Message-ID: <200812181624.33796.imriz@inter.net.il> On Thursday 18 December 2008 16:18:54 Jesse Thompson wrote: > Imri Zvik wrote: > > We would like to keep a match for at least a week ~ > > I'm curious... what's your reasoning? Some users complained about the delay - if the triplet is kept for more time, the less it would be noticeable. I'm an ISP, so I need to actually listen to what my users has to say :) > > Jesse From eino at utu.fi Fri Dec 19 14:17:57 2008 From: eino at utu.fi (Eino Tuominen) Date: Fri, 19 Dec 2008 14:17:57 +0200 Subject: [Gross] false positives In-Reply-To: <494A5A00.1030209@doit.wisc.edu> References: <200812161326.26177.imriz@bsd.org.il> <200812181024.11902.imriz@bsd.org.il> <494A0F06.7040300@utu.fi> <200812181337.22452.imriz@bsd.org.il> <494A5A00.1030209@doit.wisc.edu> Message-ID: <494B90F5.6010001@utu.fi> Jesse Thompson wrote: > >> Well, a problem reproduced is a problem half solved ;) >> (Should I reopen Issue 78?) >> >> Thanks for the help! > > Thanks for persisting. That bug (assuming it's the same one) was a pain > in the ass for us. I hope Eino can resolve it. Hi, I found and fixed the bug. There was some parentheses missing which caused and integer overflow which caused grossd to reserve too little memory for filters. Fixed version in the svn gross-1.0 branch. The problem was triggered iff (number_buffers + 1) * 2^filter_bits > 2^31 -- Eino Tuominen From imriz at bsd.org.il Mon Dec 22 10:52:54 2008 From: imriz at bsd.org.il (Imri Zvik) Date: Mon, 22 Dec 2008 10:52:54 +0200 Subject: [Gross] false positives In-Reply-To: <200812221049.54057.imriz@bsd.org.il> References: <200812161326.26177.imriz@bsd.org.il> <494B90F5.6010001@utu.fi> <200812221049.54057.imriz@bsd.org.il> Message-ID: <200812221052.54218.imriz@bsd.org.il> Emm, Ignore my last email - it seems I some how got the old version of src/srvutils.c probably because I was getting the trunk and the commit was on the 1.0 branch :P On Monday 22 December 2008 10:49:54 Imri Zvik wrote: > Hi :) > > On Friday 19 December 2008 14:17:57 Eino Tuominen wrote: > > Jesse Thompson wrote: > > >> Well, a problem reproduced is a problem half solved ;) > > >> (Should I reopen Issue 78?) > > >> > > >> Thanks for the help! > > > > > > Thanks for persisting. That bug (assuming it's the same one) was a > > > pain in the ass for us. I hope Eino can resolve it. > > > > Hi, > > > > I found and fixed the bug. There was some parentheses missing which > > caused and integer overflow which caused grossd to reserve too little > > memory for filters. Fixed version in the svn gross-1.0 branch. > > > > The problem was triggered iff > > > > (number_buffers + 1) * 2^filter_bits > 2^31 > > still crashing... > > Mon Dec 22 10:46:10 2008 #40807960: a=greylist d=0 w=0 c=111.111.11.111 > s=xxxxx at xxxxx.com.au r=greg at zzzzz.com h= > Mon Dec 22 10:46:10 2008 #40807960: sjsms_connection returning > Mon Dec 22 10:46:10 2008 #40100960: passing message from inq to outq > Mon Dec 22 10:46:10 2008 #40100960: waiting for messages > Mon Dec 22 10:46:10 2008 #40100960: passing message from inq to outq > Mon Dec 22 10:46:10 2008 #40100960: waiting for messages > Mon Dec 22 10:46:10 2008 #40100960: reftime in future, sleeping for > 0.893000 seconds > Mon Dec 22 10:46:10 2008 #40201960: received update command > Segmentation fault > > SVN version from 5 minutes ago. > filter_bits = 28 From imriz at bsd.org.il Mon Dec 22 10:49:54 2008 From: imriz at bsd.org.il (Imri Zvik) Date: Mon, 22 Dec 2008 10:49:54 +0200 Subject: [Gross] false positives In-Reply-To: <494B90F5.6010001@utu.fi> References: <200812161326.26177.imriz@bsd.org.il> <494A5A00.1030209@doit.wisc.edu> <494B90F5.6010001@utu.fi> Message-ID: <200812221049.54057.imriz@bsd.org.il> Hi :) On Friday 19 December 2008 14:17:57 Eino Tuominen wrote: > Jesse Thompson wrote: > >> Well, a problem reproduced is a problem half solved ;) > >> (Should I reopen Issue 78?) > >> > >> Thanks for the help! > > > > Thanks for persisting. That bug (assuming it's the same one) was a pain > > in the ass for us. I hope Eino can resolve it. > > Hi, > > I found and fixed the bug. There was some parentheses missing which > caused and integer overflow which caused grossd to reserve too little > memory for filters. Fixed version in the svn gross-1.0 branch. > > The problem was triggered iff > > (number_buffers + 1) * 2^filter_bits > 2^31 still crashing... Mon Dec 22 10:46:10 2008 #40807960: a=greylist d=0 w=0 c=111.111.11.111 s=xxxxx at xxxxx.com.au r=greg at zzzzz.com h= Mon Dec 22 10:46:10 2008 #40807960: sjsms_connection returning Mon Dec 22 10:46:10 2008 #40100960: passing message from inq to outq Mon Dec 22 10:46:10 2008 #40100960: waiting for messages Mon Dec 22 10:46:10 2008 #40100960: passing message from inq to outq Mon Dec 22 10:46:10 2008 #40100960: waiting for messages Mon Dec 22 10:46:10 2008 #40100960: reftime in future, sleeping for 0.893000 seconds Mon Dec 22 10:46:10 2008 #40201960: received update command Segmentation fault SVN version from 5 minutes ago. filter_bits = 28 From imriz at bsd.org.il Mon Dec 22 11:01:06 2008 From: imriz at bsd.org.il (Imri Zvik) Date: Mon, 22 Dec 2008 11:01:06 +0200 Subject: [Gross] false positives In-Reply-To: <494B90F5.6010001@utu.fi> References: <200812161326.26177.imriz@bsd.org.il> <494A5A00.1030209@doit.wisc.edu> <494B90F5.6010001@utu.fi> Message-ID: <200812221101.06639.imriz@bsd.org.il> On Friday 19 December 2008 14:17:57 Eino Tuominen wrote: > Jesse Thompson wrote: > >> Well, a problem reproduced is a problem half solved ;) > >> (Should I reopen Issue 78?) > >> > >> Thanks for the help! > > > > Thanks for persisting. That bug (assuming it's the same one) was a pain > > in the ass for us. I hope Eino can resolve it. > > Hi, > > I found and fixed the bug. There was some parentheses missing which > caused and integer overflow which caused grossd to reserve too little > memory for filters. Fixed version in the svn gross-1.0 branch. > > The problem was triggered iff > > (number_buffers + 1) * 2^filter_bits > 2^31 Hi, Now (with the right branch from SVN :) ), I cannot go over 28. Anything above 28 results a crash similar to the one before.