2 Replies Latest reply on Mar 29, 2016 8:35 AM by 1336415

    Limit on bit vector size

    1336415

      Hello,

       

      Is there any limits on bit vector size? Using vector.so it is impossible to get more than 536870912 results.

      Imagine vector filled with integers with value 42. Let's search for values between 41 and 43 in arrays of size between 500E6 and 800E6:

       

      $ ./bin/dax-in-range 41 43 500000000

      0.422279 seconds to find numbers between 41 and 43 using DAX. There are 500000000 such numbers

      $ ./bin/dax-in-range 41 43 600000000

      0.579878 seconds to find numbers between 41 and 43 using DAX. There are 536870912 such numbers

      $ ./bin/dax-in-range 41 43 700000000

      0.803640 seconds to find numbers between 41 and 43 using DAX. There are 536870912 such numbers

      $ ./bin/dax-in-range 41 43 800000000

      1.014522 seconds to find numbers between 41 and 43 using DAX. There are 536870912 such numbers

      $

      I'm searching using following calls:

      bitVec = vector_in_range(valuesVec, &low, &high);

      rescount = bit_vector_count(bitVec);

       

      Is it DAX limitation or vector.so implementation?

        • 1. Re: Limit on bit vector size
          Sarkar-Oracle

          I tried to reproduce the problem using two methods, with the same logic. Both gives me correct result.

           

          1. I generate 700000000 numbers in memory, load that array into dax vector. Do the filter operation as the person give.

           

              size_t nelem = 700000000;

              int *content = (int *)malloc(nelem * sizeof(int));

              for(size_t i = 0; i < nelem; i++) {

                  content[i] = 42;

              }

           

              vector valVec = vector_load_from_array(content, nelem, INTEGER, sizeof(int));

              int low = 41, high = 43;

              vector bitVec = vector_in_range(valVec, &low, &high);

              int rescount = getVectorBitCount(bitVec);

              printf("count: %d\n", rescount);

           

           

              Output:

              count: 700000000

           

           

          2. I generate 700000000 numbers of 42 in a singleNumber.txt file and do the same logic above:

           

              vector valVec = vector_load_from_file("./singleNumber.txt", INTEGER, sizeof(int));

              int low = 41, high = 43;

              vector bitVec = vector_in_range(valVec, &low, &high);

              int rescount = getVectorBitCount(bitVec);

              printf("count: %d\n", rescount);

           

           

              Output:

              Reading Data from file...

              count: 700000000

           

          Can you try whether the above code fragment works on your system ? If you still have problem, please contact us at swisdev_help_ww_grp@oracle.com

           

          Thanks for checking out DAX.

          • 2. Re: Limit on bit vector size
            1336415

            Hello,

             

            I tried to compile your example, but there is no getVectorBitCount() in vector.h:

             

            $ grep -i getVectorBitCount ./include/Vector.h

            $ grep -i Count ./include/Vector.h

            int vector_elem_count(vector this_vector);

            int bit_vector_count(vector bit_vector);

            $

             

            I was using bit_vector_count() to count number of resulting bits.