This discussion is archived
5 Replies Latest reply: May 3, 2013 7:35 AM by 1004273 RSS

ustack in linux dtrace

1004273 Newbie
Currently Being Moderated
Hello,

I have gotten the dtrace rpm's from uln

[XXXX@vico ~]$ rpm -qa | grep dtrace
dtrace-utils-0.3.2-1.el6.x86_64
kernel-uek-dtrace-2.6.39-201.0.2.el6uek.x86_64
libdtrace-ctf-0.3.2-1.x86_64
kernel-uek-dtrace-firmware-2.6.39-201.0.2.el6uek.x86_64
kernel-uek-dtrace-devel-2.6.39-201.0.2.el6uek.x86_64
dtrace-modules-2.6.39-201.0.2.el6uek-0.3.2-1.el6.x86_64

and dtrace works for some things (once I modprobe the dtrace & systrace modules)
[XXXX@vico ~]$ sudo dtrace -l | wc
596 2382 44993

I have a dtrace program that prints out information when a process makes a syscall to connect

#!/usr/sbin/dtrace -qs

syscall::connect:entry
{
socks = (struct sockaddr*) copyin(arg1, arg2);
hport = (uint_t) socks->sa_data[0];
lport = (uint_t) socks->sa_data[1];
hport <<= 8;
port = hport + lport;

printf("%Y pid: %d name: \"%s\" ppid: %d connected to: %d.%d.%d.%d:%d\n", walltimestamp, pid, execname, ppid, socks->sa_data[2], socks->sa_data[3], socks->sa_data[4], socks->sa_data[5], port);
ustack(10);
}

I have two different programs (that I wrote for testing) connect1.c and connect2.c that call connect through a different set of functions.
connect1.c:
int main(int argc, char* argv[]){
int sock;
struct sockaddr_in server;
char buffer[BUFFSIZE];

/*Create the TCP socket */
if((sock = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP)) < 0){
perror("Unable to create socket");
exit(1);
}

/*Construct the infos necessary to connect*/
memset(&server, 0, sizeof(server)); /* initialze the memory */
server.sin_family = AF_INET; /* We are using IP */
server.sin_addr.s_addr = inet_addr("YYYY");
server.sin_port = htons(80);

/*Make the connection*/
if(connect(sock, (struct sockaddr*) &server, sizeof(server)) < 0){
perror("Unable to connect to server");
exit(1);
}

/*clean up*/
close(sock);
return 0;
}

connect2.c
int doConnection(char* address, int port){
int sock = 0;
struct sockaddr_in server;
/*Create the TCP socket */
if((sock = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP)) < 0){
perror("Unable to create socket");
exit(1);
}

/*Construct the infos necessary to connect*/
memset(&server, 0, sizeof(server)); /* initialze the memory */
server.sin_family = AF_INET; /* We are using IP */
server.sin_addr.s_addr = inet_addr(address);
server.sin_port = htons(port);

/*Make the connection*/
if(connect(sock, (struct sockaddr*) &server, sizeof(server)) < 0){
perror("Unable to connect to server");
exit(1);
}
return sock;
}

int main(int argc, char* argv[]){
int sock;
char buffer[BUFFSIZE];

sock = doConnection("YYYY", 80);

/*clean up*/
close(sock);
return 0;
}

Using gdb to put a break point on connect (the user facing libc function) in these programs verifies that they do in fact have different call stacks.
When using my dtrace program on a linux box the call stacks are the same (this is not expected):

2013 Apr 19 16:08:56 pid: 6468 name: "connect1" ppid: 3666 connected to: YYYY:80

0xffffffffa0869277
0xffffffffa087778e
0xffffffff814253cf
0xffffffff81507a0e
0xffffffff8110c09e
0xffffffff810cc877
0xffffffff8150fd42
2013 Apr 19 16:08:57 pid: 6470 name: "connect2" ppid: 3666 connected to: YYYY:80

0xffffffffa0869277
0xffffffffa087778e
0xffffffff814253cf
0xffffffff81507a0e
0xffffffff8110c09e
0xffffffff810cc877
0xffffffff8150fd42

On a solaris box the call stacks are different (as expected).
2013 Apr 19 16:12:20 pid: 2351 name: "connect1" ppid: 1829 connected to: YYYY:80

0xfee43447
0xfef5ca03
0x8050f22
0x8050d27
2013 Apr 19 16:12:21 pid: 2354 name: "connect2" ppid: 1829 connected to: YYYY:80

0xfee43447
0xfef5ca03
0x8050f3a
0x8050f8b
0x8050d4b

My questions are:
1) Why does ustack not seem to work correctly on linux?
2) Is it possible to write a more complex dtrace program for linux that would produce a correct stack trace?

Thanks!

Edited by: user6691418 on Apr 19, 2013 3:24 PM
  • 1. Re: ustack in linux dtrace
    Dude! Guru
    Currently Being Moderated
    Are you using the same compiler under Solaris and Linux with the same code optimization options?
  • 2. Re: ustack in linux dtrace
    1004273 Newbie
    Currently Being Moderated
    The compilers are relatively close they are both GCC 3.x on both systems I was sure to compile with debugging symbols. Using gdb in both linux and solaris shows identical stack traces as far as function names and call order, just different addresses. I was originally thinking that optimizations were changing things but the gdb stack traces indicate this is not the case.
    Thanks!
  • 3. Re: ustack in linux dtrace
    1004273 Newbie
    Currently Being Moderated
    Bump
    Any ideas?

    Thanks!
  • 4. Re: ustack in linux dtrace
    TommyReynolds Expert
    Currently Being Moderated
    I think what you are actually getting is the stack information for the vsyscall fast-system call springboard, invoked from a shared dynamic library. The fast system call mechanism does some freaky magic to switch into system mode and context.

    http://lwn.net/Articles/446528/

    Now, I'm not sure if that is really useful so it's probably a bug or a work-in-progress.
  • 5. Re: ustack in linux dtrace
    1004273 Newbie
    Currently Being Moderated
    Ok interesting thanks for the info.
    As I don't entirely understand let me ask a question.
    Can anyone who implemented the kernel bits for dtrace on Linux verify this idea? It seems rather useless to have a userland stack trace that is always the same no matter where is a userland application a syscall is made.
    Thanks!

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points