8 Replies Latest reply: Jun 22, 2010 2:11 PM by 807575 RSS

    bug in mpi on solaris 10?

    807736
      hello all,
      we are using SS12u1 in solaris 10 and LInux.
      Here is a sample:
        integer  :: nj, nk, ni
      
          ni = -4
      
         print *,"MPI=",mpi.id,'NI=',ni," N1=",n1,"L=", mpi%left, "R=",mpi%right
         print *," CALL MPI_Sendrecv "
         call  MPI_Sendrecv( b, 1  , MPI_DOUBLE_PRECISION , mpi%right, 2, &
                        &    c, 1  , MPI_DOUBLE_PRECISION , mpi%left , 2, &
                        & mpi%com, mpi%stat, ierr )
         print *,"FIN MPI_Sendrec MPI=", mpi%id," NI=",ni," NJ=",nj," NK=",nk,' ierr', ierr, 'b=',b,'c=',c
      Here are the results:
      on solaris:
      Sara-rene% mpif90 -o pmpim -g pmpim.f90
      Sara-rene% mpirun -np 2 pmpim
      MPI= 1 NI= -4  N1= 6 L= 0 R= -2
      MPI= 0 NI= -4  N1= 6 L= -2 R= 1
       CALL MPI_Sendrecv
       CALL MPI_Sendrecv
      FIN MPI_Sendrec MPI= 0  NI= 0  NJ= 12  NK= 13  ierr 0 b= 0.0E+0 c= -1.0
      FIN MPI_Sendrec MPI= 1  NI= 0  NJ= 12  NK= 13  ierr 0 b= 1.0 c= 0.0E+0 
      on Linux:
      MPI= 1 NI= -4  N1= 6 L= 0 R= -2
       CALL MPI_Sendrecv
      MPI= 0 NI= -4  N1= 6 L= -2 R= 1
       CALL MPI_Sendrecv
      FIN MPI_Sendrec MPI= 0  NI= -4  NJ= 12  NK= 13  ierr 0 b= 0.0E+0 c= -1.0
      FIN MPI_Sendrec MPI= 1  NI= -4  NJ= 12  NK= 13  ierr 0 b= 1.0 c= 0.0E+0
      As you can see, NI is modified after the call to MPI_Sendrecv, on solaris!
      The same program works well when written in C, so the problem arises only in Fortran and Solaris.

      We found a simple workaround, we have to compile with "-x02" and the NI var is ok!

      thanks in advance for help,

      gerard
        • 1. Re: bug in mpi on solaris 10?
          807575
          It's hard to tell what might be going on without a complete program. There are a few things you might do to narrow it down:

          1. Have you tried compiling with -xO0 or -xO1? What happens with those optimization levels?

          2. Have you tried compiling with -m64? This is the default on Linux (assuming you are running on a 64 bit system), while -m32 is the default on Solaris.

          3. Have you compared the addresses of NI, B, C, MPI%STAT, and IERR?

          4. Have you tried using dbx's memory access checking (type "check -all" at the dbx command prompt before running the program)?

          Hope this helps.
          • 2. Re: bug in mpi on solaris 10?
            807575
            Hi,

            1) When i'm using -xO0 option, it's working. eg : mpif90 -o exe -xO0 pmpi1.f90

            2) When i'm using the -m64 option, there is no problem. eg : mpif90 -o exe -m64 pmpi1.f90

            4) I didn't succeed in using dbx in parallel. I've got some strange errors, like not finding libmtsk_db on a solaris machine.

            Thanks for the hints.
            • 3. Re: bug in mpi on solaris 10?
              807575
              It seems as though the NI variable is being placed at an address that happens to be adjacent to (or even overlaps) some other variable that is legitimately being modified. For example, perhaps NI comes after C, and C is smaller than MPI_DOUBLE_PRECISION. Or perhaps NI comes after IERR, and IERR is smaller than the MPI library expects.

              Compiling with -xO0 (or higher) places variables at different addresses because it invokes a different code generator. Compiling with -m64 will also result in different variable layout. There may be some other variable that gets trashed when you use those options (it may even be a compiler generated variable).

              I think it would be prudent to spend a little more time trying to figure out what is going on with the failing case.
              • 4. Re: bug in mpi on solaris 10?
                807575
                Hello,

                Here's a small example of the problem :
                homard-jobic% mpirun -np 3 ./exe
                 AVANT_MPI= 1  nx= 5  ny= 4  nz= 3
                 AVANT_MPI= 2  nx= 5  ny= 4  nz= 3
                 APRES_MPI= 2  nx= 5  ny= 4  nz= 3
                 AVANT_MPI= 0  nx= 5  ny= 4  nz= 3
                 APRES_MPI= -1  nx= 5  ny= 4  nz= 3
                 APRES_MPI= 2  nx= 5  ny= 4  nz= 3
                Here we can see that the value of "APRES_MPI= -1" has change after the MPI call.

                We have done this example without the structure, and it's working fine. So it's related to the use of a structure inside the MPI subroutine.

                Here is the code :
                homard-jobic% cat tabo.f90 
                MODULE VMPI
                 INCLUDE 'mpif.h'
                 type varmpi 
                  integer :: nproc             ! nombre de procs utilises                
                  integer :: com      ! le communicateur                        
                  integer :: stat      ! le statut du transfert                  
                  integer :: id                ! l'id du proc en cours                   
                  integer :: dim(3)            ! Tableau dimension 1->x 2->y  3->z       
                  integer :: coord(3)          ! Carto xy du plan 1->x  2->y  3->z         
                  integer :: reord ;            ! si on veut reoordonne defautl 0         
                  integer :: per(3) ;           ! si dim peridodique default 0            
                  integer :: left, right       ! les voisins -1 si rien                  
                  integer :: above, below 
                  integer :: up , down
                end type  Varmpi 
                end module Vmpi
                
                program demo
                  use Vmpi
                  implicit none
                  type (varmpi)     :: mpi 
                
                  integer ::nx,ny,nz
                  integer ::i,j,k,m, ierr
                  integer :: npx, npy , npz ;
                  INTEGER ::    a(5,4,3), b(5,4,3), c(5,4,3)
                  npx = 3 ; npy =1 ; npz = 1;
                  nx =5
                  ny =4
                  nz =3
                  m = nx*ny*nz
                       
                  call MPI_INIT(ierr)
                  call init_mpi( mpi , npx, npy, npz)
                
                
                        do k = 1 , nz
                          do j= 1 , ny
                            do i= 1, nx
                              a(i,j,k) = (i*100 + 10*j + k)
                            b(i,j,k) =0
                            c(i,j,k) =0
                           enddo
                          enddo
                        enddo
                   print *,"AVANT_MPI=",mpi%id," nx=",nx," ny=",ny," nz=",nz
                   call  mpi_Sendrecv( a(1,1,1), m  , MPI_INTEGER , mpi%right, 2, &
                                 &        b(1,1,1), m  , MPI_INTEGER , mpi%left , 2, &
                                 &  mpi%com, mpi%stat, ierr )
                   print *,"APRES_MPI=",mpi%id," nx=",nx," ny=",ny," nz=",nz
                   call  mpi_Sendrecv( a(1,1,1) , m  , MPI_INTEGER , mpi%left , 3, &
                                    &        c(1,1,1) , m  , MPI_INTEGER , mpi%right, 3, &
                                  & mpi%com, mpi%stat, ierr )
                 if ( mpi%id == 1 ) then
                    do k=1,nz
                      do j= 1,ny
                            do i=1,nx
                             write (*,'("(",i1,",",i1,",",i1,")=",100i4)')i,j,k,b(i,j,k),a(i,j,k),c(i,j,k)
                         enddo
                        enddo
                    enddo
                 endif
                 call MPI_Finalize(ierr)
                 end  program demo
                 
                
                subroutine  init_mpi ( mx, npx , npy, npz )
                  use Vmpi
                  implicit none
                  integer , intent(inout) :: npx, npy, npz
                  type (varmpi) , intent (inout) :: mx
                
                  integer ::ierr=0
                  
                  call MPI_COMM_SIZE(mpi_COMM_WORLD, mx%nproc,ierr)
                  call MPI_COMM_RANK(mpi_COMM_WORLD, mx%id   , ierr)
                
                
                   mx%dim(1) = npx ; mx%dim(2) = npy ; mx%dim(3) = npz ;
                   mx%per(1) = 0   ; mx%per(2 )= 0   ; mx%per(3) = 0 ;
                   call MPI_DIMS_CREATE( mx%nproc, 3, mx%dim, ierr ) 
                   mx%reord = 0 ;  
                  call MPI_CART_CREATE( MPI_COMM_WORLD, 3, mx%dim, mx%per, mx%reord ,mx%com, ierr)
                  call MPI_Comm_rank(mx%com, mx%id, ierr)
                  call MPI_Cart_coords(mx%com, mx%id, 3, mx%coord, ierr);
                  call MPI_Cart_shift(mx%com, 0, 1, mx%left  , mx%right, ierr)
                  call MPI_Cart_shift(mx%com, 1, 1, mx%below , mx%above, ierr)
                  call MPI_Cart_shift(mx%com, 2, 1, mx%down  , mx%up   , ierr)
                  return
                
                end subroutine init_MPI
                • 5. Re: bug in mpi on solaris 10?
                  807575
                  The problem is with the definition of varmpi%stat. The MPI specification says that stat should be an integer array, with MPI_STATUS_SIZE elements. You have made it a scalar.
                  • 6. Re: bug in mpi on solaris 10?
                    807575
                    I'm not it's the problem.
                    Here is the code without structure, and it's working fine :
                      program demo
                      implicit none
                      INCLUDE 'mpif.h'
                    
                      integer :: nproc             ! nombre de procs utilises                
                      integer:: stat      ! le statut du transfert                  
                      integer :: com      ! le communicateur         
                      integer :: id                ! l'id du proc en cours                   
                      integer :: dim(3)            ! Tableau dimension 1->x 2->y  3->z       
                      integer :: coord(3)          ! Carto xy du plan 1->x  2->y  3->z         
                      integer :: reord ;            ! si on veut reoordonne defautl 0         
                      integer :: per(3) ;           ! si dim peridodique default 0            
                      integer :: left, right       ! les voisins -1 si rien                  
                      integer :: above, below 
                      integer :: up , down
                    
                      integer ::nx,ny,nz
                      integer ::i,j,k,m, ierr
                      integer :: npx, npy , npz ;
                      INTEGER ::    a(5,4,3), b(5,4,3), c(5,4,3)
                      
                      npx =3 ; npy =1 ; npz = 1;
                      nx =5
                      ny =4
                      nz =3
                      m = nx*ny*nz
                      
                      call MPI_INIT(ierr)
                      call MPI_COMM_SIZE(MPI_COMM_WORLD, nproc, ierr)
                      call MPI_COMM_RANK(MPI_COMM_WORLD, id   , ierr)
                      dim(1) = npx ; dim(2) = npy ; dim(3) = npz ;
                      per(1) = 0   ; per(2 )= 0   ; per(3) = 0 ;
                      reord = 0 
                    
                      call MPI_DIMS_CREATE( nproc, 3, dim, ierr ) 
                      call MPI_CART_CREATE( MPI_COMM_WORLD, 3, dim, per, reord ,com, ierr)
                      call MPI_COMM_RANK(com, id, ierr)
                      call MPI_CART_COORDS(com, id, 3, coord, ierr);
                      call MPI_CART_SHIFT(com, 0, 1, left  , right, ierr)
                      call MPI_CART_SHIFT(com, 1, 1, below , above, ierr)
                      call MPI_CART_SHIFT(com, 2, 1, down  , up   , ierr)
                            do k = 1 , nz
                               do j= 1 , ny
                                do i= 1, nx
                                  a(i,j,k) = (i*100 + 10*j + k)
                                b(i,j,k) =0
                                c(i,j,k) =0
                               enddo
                              enddo
                            enddo
                       print *,"AVANT_MPI=",id," nx=",nx," ny=",ny," nz=",nz
                      call  mpi_Sendrecv( a(1,1,1), m  , MPI_INTEGER , right, 2, &
                                     &    b(1,1,1), m  , MPI_INTEGER , left , 2,  com, stat, ierr )
                       print *,"APRES_MPI=",id," nx=",nx," ny=",ny," nz=",nz
                      call  mpi_Sendrecv( a(1,1,1), m  , MPI_INTEGER , left , 3, &
                                       &    c(1,1,1) , m  , MPI_INTEGER , right, 3,  com, stat, ierr )
                     if ( id == 1 ) then
                        do k=1,nz
                          do j= 1,ny
                                do i=1,nx
                                 write (*,'("(",i1,",",i1,",",i1,")=",100i4)')i,j,k,b(i,j,k),a(i,j,k),c(i,j,k)
                             enddo
                            enddo
                        enddo
                     endif
                     call MPI_Finalize(ierr)
                     end  program demo
                    • 7. Re: bug in mpi on solaris 10?
                      807575
                      Try adding
                      common // stat, id
                      I think you'll see the same behavior as with the version that uses the structure.

                      Or try changing the definition of stat in the structure version, and I think you'll see the problem go away.
                      • 8. Re: bug in mpi on solaris 10?
                        807575
                        You're right. I changed the definition of mpi%stat, as it's written in the doc, and it's working fine.

                        Many thanks for your help.

                        Yann