Forum Stats

  • 3,727,624 Users
  • 2,245,419 Discussions
  • 7,852,894 Comments

Discussions

Limit on bit vector size

1336415
1336415 Member Posts: 12

Hello,

Is there any limits on bit vector size? Using vector.so it is impossible to get more than 536870912 results.

Imagine vector filled with integers with value 42. Let's search for values between 41 and 43 in arrays of size between 500E6 and 800E6:

$ ./bin/dax-in-range 41 43 500000000
0.422279 seconds to find numbers between 41 and 43 using DAX. There are 500000000 such numbers
$ ./bin/dax-in-range 41 43 600000000
0.579878 seconds to find numbers between 41 and 43 using DAX. There are 536870912 such numbers
$ ./bin/dax-in-range 41 43 700000000
0.803640 seconds to find numbers between 41 and 43 using DAX. There are 536870912 such numbers
$ ./bin/dax-in-range 41 43 800000000
1.014522 seconds to find numbers between 41 and 43 using DAX. There are 536870912 such numbers
$

I'm searching using following calls:

bitVec = vector_in_range(valuesVec, &low, &high);
rescount = bit_vector_count(bitVec);

Is it DAX limitation or vector.so implementation?

Answers

  • Sarkar-Oracle
    Sarkar-Oracle Member Posts: 6
    edited March 2016

    I tried to reproduce the problem using two methods, with the same logic. Both gives me correct result.

    1. I generate 700000000 numbers in memory, load that array into dax vector. Do the filter operation as the person give.

        size_t nelem = 700000000;

        int *content = (int *)malloc(nelem * sizeof(int));

        for(size_t i = 0; i < nelem; i++) {

            content[i] = 42;

        }

        vector valVec = vector_load_from_array(content, nelem, INTEGER, sizeof(int));

        int low = 41, high = 43;

        vector bitVec = vector_in_range(valVec, &low, &high);

        int rescount = getVectorBitCount(bitVec);

        printf("count: %d\n", rescount);

        Output:

        count: 700000000

    2. I generate 700000000 numbers of 42 in a singleNumber.txt file and do the same logic above:

        vector valVec = vector_load_from_file("./singleNumber.txt", INTEGER, sizeof(int));

        int low = 41, high = 43;

        vector bitVec = vector_in_range(valVec, &low, &high);

        int rescount = getVectorBitCount(bitVec);

        printf("count: %d\n", rescount);

        Output:

        Reading Data from file...

        count: 700000000

    Can you try whether the above code fragment works on your system ? If you still have problem, please contact us at [email protected]

    Thanks for checking out DAX.

  • 1336415
    1336415 Member Posts: 12
    edited March 2016

    Hello,

    I tried to compile your example, but there is no getVectorBitCount() in vector.h:

    $ grep -i getVectorBitCount ./include/Vector.h

    $ grep -i Count ./include/Vector.h

    int vector_elem_count(vector this_vector);

    int bit_vector_count(vector bit_vector);

    $

    I was using bit_vector_count() to count number of resulting bits.

Sign In or Register to comment.