This discussion is archived
8 Replies Latest reply: Feb 10, 2013 11:40 AM by 801338 RSS

excessive load time building large JList

801338 Newbie
Currently Being Moderated
I have an [url http://r0k.us/graphics/SIHwheel.html]application using a JList. The size is variable, and can be reloaded with a different list of things. Up to now, the max list size has been about 1600, and (re)initializations have been quick, a matter of a few seconds for the 1600-element list.

Now I'd like to work with larger lists, one with over 4,000 elements, and one with over 60,000. The 4,000 element one loads in about 20 seconds (significantly more than double the 1,600-element list). The 60,000-element one takes many hours -- so many I have not allowed it to complete. The problem is, the more elements there are, the longer it takes to add a new element.

Here is a simplified version of the list re-initialization. Note that "model" is the data represented by the list.
class simpler
{
    final DefaultListModel model = new DefaultListModel();
    private JList               list;
// following line in GUI initialization
//        list = new JList(model);

    private void getSortedNames(int sortCmd)
    {
        int howbig = 50000;
        ColorName worker;
        String hexName[];

        // clear-out old elements
        list.clearSelection();
        model.removeAllElements();  // without this line, even small lists take long time
        model.ensureCapacity(howbig);  // just added hoping it would help.  It doesn't.

        for (int i = 0; i < howbig; i++)
        {
if ((i % 16) == 1) System.out.print(i + " ");
            hexName = new String[2];
//            worker = ColorName.findColorName(i);
//            hexName[0] = worker.getHex();
//            hexName[1] = worker.getName();
hexName[0] = Integer.toString(i);
hexName[1] = Integer.toString(i + 1);
            model.addElement(hexName);
        }
    }
}
The lines within the loop which start on the first column are debug stuff. I wasn't sure whether the slowdown was due to findColorName() or addElement(), so I took findColorName() out of play. No effect on speed, so I'm pretty sure it is within the addElement() voodoo where the progressive slowdown problem lies.

If I do not create a new hexName[] each iteration, the process is extremely quick. Of course, that creates 50,000 list items of the same instance of the same data.

After studying [url http://docs.oracle.com/javase/1.4.2/docs/api/javax/swing/DefaultListModel.html]Class DefaultListModel, I tried replacing model.addElements(hexName) with both model.set(i, hexName) and model.setElementAt(hexName, i) . Both led to runtime errors, with the Jlist crashing. (In some cases the rest of the program would keep going.) I'm not sure what the difference is between the set() and setElementAt() is, but neither worked. I was hoping with the whole JList pre-allocated ahead of time, it would be quicker to set by index rather than add-at-end. Maybe it would be if the operation could complete without crashing.

Does anyone have an idea how to speed this up?
  • 1. Re: excessive load time building large JList
    sabre150 Expert
    Currently Being Moderated
    I'm very uncomfortable with users being presented with 60,000 element list and would look for an alternative approach.

    Having said that I would expect you to be creating a ListModel derived from AbstractListModel. Using this you will not have an event being sent every time you add an element; you can fire just one event when the list is fully populated. Also, you can directly wrap the data structure that you are currently copying from so no copying is actually necessary.

    It is very common for people to copy data from one data structure to another when all they really need to do is to write a wrapper class that provides the data in the right format without needing any copying. If, as would seem to be he case in your case, sorting is required one creates a sorted index (or array of references) to the original data then the only new data required is as an array of indexes or references with values pointing to the unsorted data.

    You seem to be generating the list so as to get a String representation of the data. This should not be done in the model; it should be done with a renderer. Once again this eliminates the need to copy the data.
  • 2. Re: excessive load time building large JList
    801338 Newbie
    Currently Being Moderated
    Sabre, thank you for the response. I will look into the optimizations you suggest. It turns out, there is something going on that I still do not understand. There is a way to load even the very large lists very quickly. All that is needed is for the user to click the GUI of the JList, either an individual item or the slider. Once it's GUI has actually done something in addition to its initial rendering, the next list loads very quickly. My 64,460-element list loads in less than 3 seconds! Without the JList GUI being activated, I know it takes over 3 hours. (I didn't wait longer than that, so I don't know how long.)

    I will investigate whether similar results can be achieved programmatically, perhaps with a list.setSelectedIndex(i). If so, I'll just make sure to do that after every list reload.
    sabre150 wrote:
    I'm very uncomfortable with users being presented with 60,000 element list and would look for an alternative approach.
    For most uses, I would agree. My program is displaying a list of color names, with the background of each name item being the color it names. Achieving a smooth transition between each list item was the goal. The list is sortable in various ways. Having a large list overcomes the poor color space coverage of most lists of color names, so that every color has lots of nearby neighbors.
    ...
    You seem to be generating the list so as to get a String representation of the data. This should not be done in the model; it should be done with a renderer. Once again this eliminates the need to copy the data.
    Hmm, what would I put in the model, if anything?

    There is a renderer which operates on the model's strings to set the background color of each item and the displayed color name. I guess you are saying I can bypass the model and render directly. I'll need to study to figure out how to handle reloadable lists of different sizes. Currently the relationship between model and list allow me to be ignore a lot of the inner workings -- which may be the very reason this problem cropped up.
  • 3. Re: excessive load time building large JList
    EJP Guru
    Currently Being Moderated
    If you have a JList with 1600 elements, let alone 64k, you have a major usability problem, which is far more important than any associated performance problem. Even when you get the thing rendered, you will be sitting there for hours if not weeks while the user looks through it. You need to reconsider this completely.
  • 4. Re: excessive load time building large JList
    sabre150 Expert
    Currently Being Moderated
    RichF wrote:
    Sabre, thank you for the response. I will look into the optimizations you suggest. It turns out, there is something going on that I still do not understand. There is a way to load even the very large lists very quickly. All that is needed is for the user to click the GUI of the JList, either an individual item or the slider. Once it's GUI has actually done something in addition to its initial rendering, the next list loads very quickly. My 64,460-element list loads in less than 3 seconds! Without the JList GUI being activated, I know it takes over 3 hours. (I didn't wait longer than that, so I don't know how long.)
    Each time you add an item to the list an event is sent and the list re-rendered! As I have already pointed out you do better extending AbstractLisModel and then firing just one event!

    >
    I will investigate whether similar results can be achieved programmatically, perhaps with a list.setSelectedIndex(i). If so, I'll just make sure to do that after every list reload.
    sabre150 wrote:
    I'm very uncomfortable with users being presented with 60,000 element list and would look for an alternative approach.
    For most uses, I would agree. My program is displaying a list of color names, with the background of each name item being the color it names.
    60,000 different colour names! Come off it! Who can even remember 60,000 different colour names yet alone tell the difference between them? Even if someone can resolve 60,000 different colours I bet that monitor differences completely mask the differences.
    Achieving a smooth transition between each list item was the goal.
    I don't understand this in relation to needing 60,000 different colours!
    The list is sortable in various ways.
    I don;t understand this if you are trying to get a "smooth transition" ! Surely there is only one sort order that gives a "smooth transition".
    Having a large list overcomes the poor color space coverage of most lists of color names, so that every color has lots of nearby neighbors.
    ...
    You seem to be generating the list so as to get a String representation of the data. This should not be done in the model; it should be done with a renderer. Once again this eliminates the need to copy the data.
    Hmm, what would I put in the model, if anything?
    Presumably you have a colour value and a colour name. Surely these pairs are what you put in the model!

    >
    There is a renderer which operates on the model's strings to set the background color of each item and the displayed color name. I guess you are saying I can bypass the model and render directly.
    No - you always need a model but you don't need a different model for each possible rendering!
    I'll need to study to figure out how to handle reloadable lists of different sizes.
    Trivial if as I suggested you wrap your data in an class derived from AbstractListModel. Normally on just changes one reference in the derived class so that it points to a different set of data and then fires one event.
    Currently the relationship between model and list allow me to be ignore a lot of the inner workings -- which may be the very reason this problem cropped up.
    I don't understand this.
  • 5. Re: excessive load time building large JList
    801338 Newbie
    Currently Being Moderated
    EJP wrote:
    If you have a JList with 1600 elements, let alone 64k, you have a major usability problem...
    You need to reconsider this completely.
    My user isn't choosing something concrete, such as a breed of dog or national currency. My [url http://r0k.us/graphics/SIHwheel.html]application is the rare case of the tail wagging the dog, so to speak. The list was added three years ago to the program which first launched 10 years prior to that. Just as often as not, the user might select a color from the wheel, and find it in the current list, or they might use the list in the "normal" manner to show where a color is on the wheel.

    Those uses together are like 1/3 of the actual utility offered. The colored list can also be used in a unique way, to see the visible relationship of colors. The program offers various sorts to modify this relationship, which was hard to discern with the spotty color coverage of most lists of color names. Think about it like this: a color space is a 3-dimensional realm, whereas a list is 1-dimensional. How does one best map 3-d to 1-d, and know that the mapping is valid?

    That all said, I have not yet decided whether to add the large list to the pre-existing set of "color name dictionaries" (CNDs). I generated it out of my own curiosity. But the text file is huge, 1.8 million bytes or 300k zipped. The current .jar file is only 110k, including all the CNDs up to now. Does it makes sense to include it? I don't yet know.
  • 6. Re: excessive load time building large JList
    801338 Newbie
    Currently Being Moderated
    Sabre, thank you again. I have some studying to do.
    60,000 different colour names! Come off it! Who can even remember 60,000 different colour names yet alone tell the difference between them?
    {Sigh} Have you gone to the [url http://r0k.us/graphics/sihImages.html]Interactive Color Wheel and played with it? It is not a color memorization tool or even a color picker. Oh, it could be used for either, but as a side effect. Try different sorts, scroll the list up and down. Heck, load a different color list and repeat.
    Even if someone can resolve 60,000 different colours I bet that monitor differences completely mask the differences.
    That someone is you. Watch the "hueborhood" as you play around. It is the small square of colors just to left and near the bottom of the intensity gauge. It shows both the selected color and the available colors nearest it from 3 dimensions. The fact that you can distinguish them means that you have better color acuity than 60k colors map. And the monitor color resolution is much higher than 60k.
    Currently the relationship between model and list allow me to be ignore a lot of the inner workings -- which may be the very reason this problem cropped up.
    I don't understand this.
    It means I started with example code and learned what I needed to get it working according to design. "What I needed" was shy of full understanding. So it is not really surprising that populating a very large list uncovered weaknesses in implementation.

    -- Rich
  • 7. Re: excessive load time building large JList
    sabre150 Expert
    Currently Being Moderated
    I started trying to reply point by point to your previous response but have come to the conclusion I really don't understand what you are trying to do. I'm on shaky ground when talking of resolving 60,000 colours so I can't continue with that argument. I do know that using a list of length around 60,000 goes against all my GUI experience and everything I have read on GUI usability. I try to limit combo boxes to no more that 15 items and lists to no more than 60 items and would never ever ever consider even 1,000 items never mind 60,000 !

    For JComboBox, JLists and JTables one should rarely add one item at a time especially if the length is significant. For every item added an event is sent to all listeners. If you add 60,000 items you get 60,000 events but only one is actually required! Think in terms of wrapper classes that implement the desired models. Think about having multiple renderers that render a given entry in the model in various ways. Think about sorting without actually creating copies or sorting the actual objects; indirect sort is very very powerful. Think about models wrapping other models and responding to events sent by these wrapped models.

    For one customer I initially read from a database into an event driven list of tuples. I have several TableModels used to show different views of this data each of which responds to the events generated by the original list. When the original list is updated by the user all the other models receive the events that indicate the change and update anything needed before notifying the GUI component that it needs to update the display. Each JTable has a set of specific renderers. Other than in the renderers I never copy the data or create string versions of the data to be displayed.
  • 8. Re: excessive load time building large JList
    801338 Newbie
    Currently Being Moderated
    sabre150 wrote:
    ... I do know that using a list of length around 60,000 goes against all my GUI experience and everything I have read on GUI usability. ... and would never ever ever consider even 1,000 items never mind 60,000 !
    Don't think of it as just a list. It is also a color ramp. More colors == smoother ramp.

    (I wonder if anyone ever told [url https://kuler.adobe.com/#create/fromacolor]these guys they have too many damn colors. ;) Admittedly they don't display many at once, but our goals are different.)
    For JComboBox, JLists and JTables one should rarely add one item at a time especially if the length is significant. ...
    Thank you for the advice. The problem is solved within the current implementation, but I'll likely re-implement using a better technique.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points