You need to know more than just the vertices. For each face of the polyhedron, you need to which vertices form the corners of the face and in which order they occur. And the direction of that ordering should be consistent between faces. The usual way to organize them is by the right-hand rule (the left-hand rule would work about as well, but the right-hand rule is the common choice). That is, if you point the thumb of your right-hand in the direction perpendicular to the face and pointing outside the polyhedron, then the order in which the vertices are listed should be in the direction that the fingers of your right hand curl.
With this information, finding the volume is a straight-forward calculation, and one commonly performed. Many analyses of object designs in industry are obtained by approximating the shape of the object with a "triangulation" of the surface: a polyhedron with many small triangular faces approximating the surface of the object. Calculating the volume of that polyhedron is the simplest of the many calculations this is used for.
The trick of it is easily described, though it takes a bit of work to accomplish:
- Convert all the faces to triangles. If you have any non-triangular faces, choose a vertex, and add diagonals to each of the other non-neighboring vertices of the face, breaking the face into triangles. The vertices for each of the triangles should once again to ordered by the right-hand-rule.
- Each triangle forms a tetrahedron with the origin. Calculate the signed volumes of these tetrahedra. The volume should be positive if the origin is on the same side of the triangular face as the polyhedron, and negative if the origin is on the opposite side of that face. This sounds complex, but if you've ordered the vertices by the right-hand rule, it is actually easy. If $v_1, v_2, v_3$ are the three vertices of the face in order, then the signed volume of the tetrahedron is $$V_T = \frac16v_1 \cdot (v_2 \times v_3)$$
- Sum the signed volumes of the tetrahedra for each of the faces. The sum will be the volume of the polyhedron.
This works no matter what the shape of the polyhedron or the location of the origin. The positive-volumes occur where the tetrahedron is on the same side of the face as the polyhedron, so it includes volume inside the polyhedron. That tetrahedron may extend outside the polyhedron, but when it does, it passes through other faces, and the tetrahedra for those faces will have negative volume and thus subtract off the portions of the positive tetrahedron that lie outside.