How is pipe syntax implemented in apache beam?

Question

I am now studying apache beam and out of curiosity, I would like to ask below question.

In advance, I have read below documents and threads.

https://beam.apache.org/documentation/programming-guide/#applying-transforms

I understand that pipe(|) is the python version of .apply of java. However, I am curious to know how does python interpret __or__ operator as processor which processes each pcollection element that goes through from left to right.

I appreciate if someone could educate me and point me the reference of code.

Thanks, Yu

I did a Google search on "python operator overloading" and found a bunch of good references that seem likely. Searching the Github repository, it looks likely that this may be the actual code: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/ptransform.py#L470 — Kolban, Apr 23 '19 at 04:43

score 0 · Accepted Answer · answered Apr 25 '19 at 06:47

I would like to mark @Kolban 's reply as answer.

I did a Google search on "python operator overloading" and found a bunch of good references that seem likely. Searching the Github repository, it looks likely that this may be the actual code: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/ptransform.py#L470

score 0 · Answer 2 · answered Jan 03 '20 at 22:08

It does so via operator overloading:

def __or__(self, right):
  """Used to compose PTransforms, e.g., ptransform1 | ptransform2."""
  if isinstance(right, PTransform):
    return _ChainedPTransform(self, right)
  return NotImplemented

Piping (|) is used to compose PTransforms, e.g., ptransform1 | ptransform2.

How is pipe syntax implemented in apache beam?

2 Answers2