[Gross] Weighted checks in production

[Jesse Thompson]

Eino Tuominen wrote:
> Eino Tuominen wrote:
>> Eino Tuominen wrote:
>>> Hi all,
>>>
>>> I have now weighted checks version in production in one node:
>>>
>>> smtp01:gross# ps -e -o etime,time,vsz,rss,pcpu,args | grep grossd
>>>     03:50:08        6:08 32408 31776  0.6 sbin/grossd
>>>
>>> Trust: 9959 Match: 46 Greylist: 3643 Block: 39232 Queries/sec: 3.823019
>>>
>>> Its now 6:30 pm here, so ham is still coming in. I uploaded a new 
>>> development snapshot on Google, play with it!
>>>
>>> Oh, I now made these ports as defaults: 5525 gross; 5522 status; 5524 
>>> sync. This time I checked IANA port list first!-)
>> Darn, I managed to upload wrong version. Now check_blocker is broken in 
>> the gross-svn-232.tar.gz. I'll fix it ASAP. Otherwise it should be stable.
>>
>> I'll be back,
> 
> First night has passed, here are the statistics of one node:
> 
> Trust: 15166 Match: 233 Greylist: 8582 Block: 120463
> 
> Quite amazing?!

Cool!  Nice work, as always.  I will try this out soon, I hope.


> I have now uploaded gross-svn-236.tar.gz, it's running now in production 
> here. I couldn't get check_blocker to reuse connections as Sophos' 
> blockerd keeps shutting every connection down right after the answer. 
> And, now there's watchdog to see if blocker threads get stuck.

Was this a bug?  I can't remember the details.


> Jesse, you threw an idea some time ago about skipping first n rotating 
> filters when calculating the effective filter. I've been pondering your 
> idea and I think it could be of use now. It might be a good idea to hold 
> a triplet as long as few hours at greylisted state, as it would give 
> Puremessage and alike time to update their filters and let the spammers 
> id get spread to another RBL's as well. What do you think?

Yeah, I think that this is the most efficient way to enable longer 
greylisted states.  From what I understand, the current method of 
setting a greylist time period keeps the entry in memory and waits X 
seconds before updating the filter.  This seems to me like it wouldn't 
scale well to larger time values.


> The original problem was that resolution of filters is so poor (1 hour 
> with default config). But now that most spam gets blocked and only a 
> portion gets greylisted there is a lot of room in the filters. We could 
> easily afford more rotating buffers skipping a few first of them.

Yeah, if the filter resolution is 1 hour, and you tell gross to ignore 
the first filter, it will mean that the triplets will be in the greylist 
state for 0 < X < 1 hour.  Otherwise you could tell gross to ignore the 
first 2 filters, which would be 1 hour < X < 2 hours.  That seems too long.

I'd probably want to first try setting the filter resolution to 30 
minutes and then ignore the first 2 filters, so the triplets will be in 
the greylist state for 30 min < X < 1 hour.  That still might be a 
little too aggressive though.  So maybe 15 minute filters, ignoring the 
first 2: 15 min < X < 30 min?

Thoughts?

Jesse


-- 
   Jesse Thompson
   Email/IM: jesse.thompson at doit.wisc.edu
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3340 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.utu.fi/pipermail/gross/attachments/20080328/34d691ca/attachment.bin>