There's a simple solution which takes advantage of the fact that semaphores naturally allow controlling access to a resource with multiple instances. Model the exclusion as a resource with two instances. A requires both instances while B and C each require one. When A starts, it grabs both instances, preventing B and C from starting. When B or C starts, it grabs one instance, which allows the other to start concurrently, but blocks A.
A: B: C:
V(S, 2) V(S, 1) V(S, 1)
… … …
P(S, 2) P(S, 1) P(S, 1)
This solution works even if the processes run multiple times. A critical ingredient that makes it work is that A atomically excludes both B and C from running.
If you treat B and C's resources as distinguishable, with A requiring both, you run into a classical problem, which is the order of grabbing locks. This problem is similar to the dining philosophers problem. When A starts, it either grabs the B resource first, or the C resource first. Let's say A grabs B, and then tries to grab C. If C is not available, what should happen? If A keeps the B resource then it prevents process B from starting. So A must release B and try again. Since A knows that C is busy, it can wait on C first. In this particular exercise, where the processes only run once, this works. Note that this requires semaphores with a non-blocking “try” operation.
A: B: C:
lock(B) lock(B) lock(C)
if try_lock(C): … …
… release(B) release(C)
release(C)
release(B)
else:
release(B)
lock(C)
lock(B)
…
release(B)
release(C)
If B and C could run multiple times, this strategy wouldn't work: A must never keep one lock while waiting for the other, because that prevents the corresponding process from running again. In this case, I think there's no wait-free solution with only binary semaphores.