7

What are the differences with sparse and dense matrices in practice, so as to offer some insight to new learners on a more intuitive level.

Obviously everyone knows about the dictionary definition of sparse and dense matrices (a definition based on the portion of zero/non-zero elements)

But why are they so important from a mathematical application/optimization/problem solving point of view?

  • Is it that a lot of neat algorithms are defined such that they can only be operated on a problem if it satisfies such and such criteria, and some guy just proved that sparse | dense matrices tends to satisfy the aforementioned criteria really well

  • Or is it to do with the limited amount of computer memory available in real life, and that we must somehow "compress" matrices for faster computation - as such sparse matrices would be more desired

  • Or is it just a fuzzy guideline word that mathematicians use, as opposed to strict criterion fulfilling definitions that imply X properties about the matrices (e.g. make sure the matrix is sparse and not dense because too many elements/variables too long to compute - or something to that nature?)

TLDR Question:

Is the only major difference as a result of computational limitation and resource savings or are there fundamental mathematical differences between the two that make one uniquely operable and the other not

TLDR Answer:

So essentially it revolves around our ability to compute something. so there really isn't some "fundamental" difference (like the difference between the first derivative or a second derivative of a function). but its just a thing that rose out of technical limitations in real life during computation.

AlanSTACK
  • 4,185
  • 1
    For example, matrix-vector multiplication takes $O(n^2)$ time on dense matrices but $O(n)$ time on (many classes of) sparse matrices. I would call that both a resource saving and a fundamental mathematical difference. –  Jan 30 '16 at 04:21

1 Answers1

10

It's not really about the matrices. It's about how the cost of certain algorithms, data structures, procedures, etc. relate to the size of the matrix and the number of non-zero elements.

For example, if a data structure is designed for storing sparse matrices, it means that it can store a matrix in space proportional to the number of non-zero elements. Simpler data structures for storing matrices (like 2-dimensional arrays) take space proportional to the size of the matrix.

If you're working with a sparse matrix data structure, it's probably because your matrix is large, but sparse, and you rely on the sparseness of the matrix to meet your requirements for the size of the data structure in memory.

Similarly, if an algorithm is designed to work with sparse matrices, then its cost is defined in terms of the number of non-zero elements in the matrix rather than the matrix's size. If you're using an algorithm like that, them again it's probably because your matrix is large, but sparse, and you rely on the sparseness of the matrix to meet your requirements for the speed of the computation.

So, in summary, there is no specific density at which the matrix changes from sparse to dense. When you start finding it useful to use data structures and algorithms that are optimized for sparse matrices, then you start thinking about the costs in terms of the number of non-zero elements, and at that point you are using sparse matrices.

ALSO, a class of matrices (e.g, diagonal, tridiagonal, etc.) is generally called sparse if the number of non-zero elements is proportional to the number of rows or columns instead of rows*columns. This gives sparse matrix algorithms an advantage in computational complexity (big O), meaning that sparse matrix algorithms will always perform better on sufficiently large matrices in that class.

Matt Timmermans
  • 448
  • so essentially it revolves around our ability to compute something.

    so there really isnt some "fundamental" difference (like the difference between the first derivative or a second derivative of a function).

    but its just a thing that rose out of technical limitations in real life during computation.

     – AlanSTACK Jan 30 '16 at 05:16
  • 4
    That is right. If you think about a $100 \times 100$ matrix, it has $10,000$ entries. If I just store it as an array, it takes that much room. If most of the entries are zero, I could store (row, column, value) for each non-zero entry. That takes three words per non-zero entry, but will be much more efficient for a sparse matrix. This would seem to imply that this would be desirable up to a density of $1/3$, but many of the operations become harder with this irregular storage, so you need it rather sparser to come out ahead. – Ross Millikan Jan 30 '16 at 05:30