[Gross] gross 1.0.2 core dumping with SJSMS 7u3 (7.3-11.01) 64-bit

[Derek Diget]

On Oct 18, 2009 at 11:39 -0400, Derek Diget wrote:
=>On Oct 18, 2009 at 08:54 +0300, Eino Tuominen wrote:
=>=>Derek Diget wrote:
=>=>> Building a new system and trying out gross.  Built c-ares and gross 1.0.2 on
=>=>> a full Solaris 10u7 (5/09) SPARC using the gcc from the install.  grossd
=>=>> runs just fine.  grosscheck.so reports it is 64-bit and ldd doesn't show
=>=>> anything missing.
=>=>> 
=>=>> Added the mappings entry in the docs and when I run an imsimta test -mapping
=>=>> in the trouble shooting doc it core dumps.  If I send in a live message,
=>=>> then tcp_smtp_server core dumps.
=>=>
=>=>First you could run pstack on the core file to make sure the crash is
=>=>happening in grosscheck().
=>
=>Yup.
=>
=>core 'mapping_test.2144' of 2144:       /opt/sun/comms/messaging64/lib/imtacli test -mapping -table ORIG_MAIL_
=> ffffffff77a0105c grosscheck (ffffffff7fffb078, ffffffff7fffb074, ffffffff7fffaa48, ffffffff7fffaa44, 0, 0) + 12c
=> ffffffff7ea5c734 mm_process_pattern (ffffffff7fffa600, 10000141, cf, ffffffff7fffa0d0, 7e, ffffffff7fff9cc8) + 298c
=> ffffffff7ea5f7fc mmc_apply_mapping (ffffffff7fffe600, ffffffff7ffff90c, a, ffffffff7ffff6fc, ffffffff7ffffa48, ffffffffffffffff) + 77c
=> 0000000100001a20 main (5, ffffffff7ffffb38, ffffffff7ffffb68, ffffffff7b54b800, ffffffff7bd00380, ffffffff7d000200) + 688
=> 000000010000119c _start (0, 0, 0, 0, 0, 0) + 17c
=>
=>
=>=>Can you send me a copy of your mapping configuration (the relevant part is
=>=>sufficient).
=>
=>ORIG_MAIL_ACCESS
=>
=>  TCP|*|*|*|*|SMTP/*|*|tcp_local|*|*|* $[IMTA_LIB:grosscheck.so,grosscheck,127.0.0.1,127.0.0.1,5525,$2,$=$8$_,$=$6$_,$=$4$_]
=>
=>
=>I have the "other" ORIG_MAIL_ACCESS lines from the example in the 
=>mappings file, but I have them currently commented out. I have tried 
=>placing grosscheck.so in /opt/gross/lib, /opt/sun/comms/messaging64/lib 
=>and changed the path accordingly in the mapping entry.  I have also 
=>tried leaving the second grossd server blank (...,127.0.0.1,,5525,...) 
=>with the same results.  
=>
=>Currently, grosscheck.so is 
=>-rwxr-xr-x   1 bin      bin        17680 Oct 15 01:25 /opt/sun/comms/messaging64/lib/grosscheck.so
=>
=>
=>=>I have not run grosscheck in 64 bit yet, is there anyone else? Jesse?

Based on some off-list pointers, I grabbed trunk/src/testgrosscheck.c 
and placed it in the gross-1.0.2/src directory.  Modifed the char arg* 
line to fit a test case.  It went from

char *arg = "127.0.0.1,,1111,127.0.0.2,foo at foo,bar at bar\0";

to

// <grossd-server-1>,<grossd-server-2>,<grossd-port>,<connecting-IP>,<to-addr>,<from-addr>,<HELO_name>
char *arg = "127.0.0.1,,5525,10.218.1.43,localrecipient at example.net,remotesender at example.com,mail.example.com\0";

Then built it using the following command:

/usr/sfw/bin/gcc -m64 -R/opt/gross/lib -o testgrosscheck testgrosscheck.c

"file testgrosscheck" looked good (64-bit).  testgrosscheck was then 
copied into /opt/gross/bin.

Running /opt/gross/bin/testgrosscheck resulted in the following output

2: $Y

And grossd showed the following (was running via "grossd -dD")

Thu Oct 22 22:46:56 2009 #b: threadpool 'sjsms' processing
Thu Oct 22 22:46:56 2009 #b: query from 127.0.0.1
Thu Oct 22 22:46:56 2009 #b: sender=remotesender at example.com
Thu Oct 22 22:46:56 2009 #b: recipient=localrecipient at example.net
Thu Oct 22 22:46:56 2009 #b: client_address=10.218.1.43
Thu Oct 22 22:46:56 2009 #b: helo_name=mail.example.com
Thu Oct 22 22:46:56 2009 #c: threadpool 'dnsbl' processing
Thu Oct 22 22:46:56 2009 #d: threadpool 'dnswl' processing
Thu Oct 22 22:46:56 2009 #c: dnsblc called: timelimit 6000
Thu Oct 22 22:46:56 2009 #d: dnsblc called: timelimit 6000
Thu Oct 22 22:46:56 2009 #e: threadpool 'rhsbl' processing
Thu Oct 22 22:46:56 2009 #e: dnsblc called: timelimit 6000
Thu Oct 22 22:46:56 2009 #c: dnsblc returning
Thu Oct 22 22:46:56 2009 #e: dnsblc returning
Thu Oct 22 22:46:56 2009 #d: dnsblc returning
Thu Oct 22 22:46:56 2009 #b: a=trust d=31 w=0 c=10.218.1.43 s=remotesender at example.com r=localrecipient at example.net h=mail.example.com
Thu Oct 22 22:46:56 2009 #b: sjsms_connection returning


So things are looking good so far. :)

It was then suggested to compile things with -DARGDEBUG.  So I did a 
"gmake clean" in the distro directory.  Ran configure using the 
following command (same as what I have been using expect for the 
additional -DARGDEBUG).

env PATH=${PATH}:/usr/sfw/bin:/usr/ccs/bin ./configure CPPFLAGS='-I/opt/c-ares/include' LDFLAGS='-L/opt/c-ares/lib' CFLAGS="-m64 -DARGDEBUG" --prefix=/opt/gross

And then 

env PATH=${PATH}:/usr/sfw/bin:/usr/ccs/bin gmake install

So far so good.

Copied the new grosscheck.so into /opt/sun/comms/messaging64/lib
and checked ownership and permissions.

Started up grossd with -dD (using the default .conf), and then ran

/opt/sun/comms/messaging64/sbin/imsimta test -mapping -noimage -debug -table ORIG_MAIL_ACCESS

with the following input string:

Input string: TCP|10.20.204.35|25|10.218.1.43|46569|SMTP/mail.example.com|MAIL|tcp_local|remotesender at example.com|l|localrecipient at example.net

23:14:44.91: Mapping 4 applied to TCP|10.20.204.35|25|10.218.1.43|46569|SMTP/mail.example.com|MAIL|tcp_local|remotesender at example.com|l|localrecipient at example.net
23:14:44.91:   Entry #1 matched, pattern "TCP|*|*|*|*|SMTP/*|*|tcp_local|*|*|*", template "$[IMTA_LIB:grosscheck.so,grosscheck,127.0.0.1,127.0.0.1,5525,$2,$=$8$_,$=$6$_,$=$4$_]", match #0.
23:14:44.91:   User routine call: IMTA_LIB:grosscheck.so\grosscheck(127.0.0.1,127.0.0.1,5525,10.218.1.43,localrecipeint at example.net,remotesender at example.com,mail.example.com) ->
Bus Error - core dumped

Not good...but did we get the ARG debug output....

Nope. :(

I did a strings against the "non-debug" and the "debug" grosscheck.so 
and the non-debug does not have "/tmp/argout", but the debug version 
does.  So I am pretty sure if it got to that point in the code it would 
have created it.

Did the above with a truss and I see the following

4278/1:         28.6614  0.0439 munmap(0xFFFFFFFF7AE00000, 32768)               = 0
4278/1:         28.6618  0.0004     Incurred fault #5, FLTACCESS  %pc = 0xFFFFFFFF77E01184
4278/1:               siginfo: SIGBUS BUS_ADRALN addr=0xFFFFFFFF7FFFB074

(I can send the truss output off-list if anyone is interested.)

Anyone have any ideas or suggestions on what I can do next to help debug 
this?

Thanks.

-- 
***********************************************************************
Derek Diget                            Office of Information Technology
Western Michigan University - Kalamazoo  Michigan  USA - www.wmich.edu/
***********************************************************************