We want Matrix, but it is too limiting...

I'm selling candles...

I want to allow people to shop by fragrance, color, and size. My thoughts are to create matrix items around the fragrance. I.e. - Fragrance is Apple with a drop down for each size; Sm, Md, Lg, etc. We would manually add different candles to certain color categories. The problem lies with trying to sell them by size only, like listing all Sm jars together, Md jars together, etc.

Does anyone have an idea how to do that? Essentially I need to sell a part from the Matrix item on its own.