For every person with with a \$10 bill we need another person with a \$5 bill that comes first. Let $k=1,2,3,\ldots,m+n$ denote the position in the queue where $k=1$. Let $n_k$ be the total number of people with \$10 dollar bills in the positions $1,2,3,...,k$ and $m_k$ be the number of people with \$5 dollar bills in these positions.
Then we need to have $m_k \geq n_k$ for every $k$ (lets call this fact (a)).
We obviously have $m_k \geq m_{k-1}$ and similarly $n_k \geq n_{k-1}$ and also $n_{m+n} = n$ and $m_{m+n} = m$. Also $m_k - m_{k-1} \leq 1$ and $n_k - n_{k-1} \leq 1$ (lets call this (c)).
We can solve this by using dynamic programming. Lets draw this as a table of states: The $k$-th antidiagonal (going from top right to bottom left, consitsting of all the states $s_{i,j}$ with $i+j=k$) shows the possible states in the $k$-th position. The column index represents $m_k$ and the the row index $m_k$. For the visualization as table I use $n=2$ and $m=3$
$$\begin{array}{c|c|c|c|c}n_k \setminus m_k & 0 & 1 & 2 & 3 \\ \hline
0 & s_{0,0} & s_{0,1} & \cdots & \\ \hline
1 & s_{1,0} & \ddots & & \\ \hline
2 & \vdots & & & \end{array}$$
Now each of the ways to arrange the queue represents a path through this from $s_{0,0}$ to $s_{n,m}$ where we are just allowed to go right or down. Due to (a) we know all subdiagonal states are not allowed
$$\begin{array}{c|c|c|c|c}n_k \setminus m_k & 0 & 1 & 2 & 3 \\ \hline
0 & & & & \\ \hline
1 & \times & & & \\ \hline
2 & \times & \times & & \end{array}$$
So now let $s_{i,j}$ count the number of ways to get to $s_{i,j}$ we have $s_{0,0} = 1$ and we are looking for $s_{n,m}$. If $i=j$ then we can just have com from the top so $s_{k,k} = s_{k-1,k}$ for all $k$, if $i=0$ we can just come from the left so $s_{0,j}=1$ for all $j$. So to compute each $s_{i,j}$ we just have to add the numbers from top and left (where possible) so if $i<j$ and $i \neq 0$ we have $s_{i,j} = s_{i-1,j} + s_{i,j-1}$, and the grid will look like so:
$$\begin{array}{c|c|c|c|c}n_k \setminus m_k & 0 & 1 & 2 & 3 \\ \hline
0 & 1 & 1 & 1 & 1 \\ \hline
1 & \times & 1 & 2 & 3\\ \hline
2 & \times & \times & 2 & 5\end{array}$$
This means to find $s_{n,m}$ you just have to evaluate the recursion given in the paragraph above.