2

I have installed Tensorflow and Metal plugin by using pip on Mac Mini 2020 M1,

$ pip3 install tensorflow-macos tensorflow-metal
$ pip3 uninstall numpy # related to https://stackoverflow.com/q/66060487/2395656 
$ pip3 install numpy==1.20.3

then I tried to list devices to see Mac GPU,

import tensorflow

d = tensorflow.config.list_physical_devices()

print(d)

It produces the error,

Init Plugin
Init Graph Optimizer
Init Kernel
Init Plugin
2021-06-10 02:20:21.128021: F tensorflow/c/experimental/stream_executor/stream_executor.cc:823] Non-OK-status: stream_executor::MultiPlatformManager::RegisterPlatform( std::move(cplatform)) status: Internal: platform is already registered with name: "METAL"

I think that Metal plugin is trying to register itself multiple times.

Please help, thanks!

Tarık Yılmaz
  • 369
  • 6
  • 10

4 Answers4

19

On my MacBook Air M1 under Monterey 12.3 and the brew-installed pythons python@3.8 and python@3.9 in their standard locations at /opt/homebrew, together with Xcode 13.3 at its standard location, after looking for hours I finally noticed the following and have this theory:

First of all I don't like to use conda or miniconda etc. (only non-python dependency I seemed to lack was hdf5 anyway). Also, I confirm that creating a venv mentioned above by tensorflow support works just fine. But if I just want to pip install tensorflow-macos and tensorflow-metal in the system locations for brewed pythons:

brew install hdf5
HDF5_DIR=/opt/homeware python3 -m pip install tensorflow-macos

If at the last stage the plugin tensorflow-metal gets installed with:

python3 -m pip install tensorflow-metal

while using a brew-ed python@3.9 from /opt/homebrew/bin/python3 at the brew system default dir: /opt/homebrew/lib/python3.9/site-packages/tensorflow-plugins

then I get the error about the METAL plugin having been already registered.

If however I install in my user directory library instead ~/Library/Python/3.9/lib/python/site-packages via

python3 -m pip install --user tensorflow-metal

then everything works. Note this concerns just the plugin, i.e. the tensorflow-macos package can still be at the brew system location.

I noticed all this, because from past Xcode Python (3.8) usage I had a user packages directory (Python Library) at ~/Library/Python/3.8/lib/Python/site-packages (only place you can add packages when using the system Python) and since it existed, the brewed python@3.8 installed everything there without me supplying --user to pip.

So I banged my head against the wall as to why brewed python@3.8 works and python@3.9 doesn't. I cleaned out all my packages from python@3.9 system directory via pip uninstall, even purged the entire system site-packages directory and still with a clean setup using the two simple pip steps it kept failing. That's because there was no user packages directory for 3.9 and everything kept going back to the brew system directory.

I am quite convinced that is not a case of some configuration or package mix clashing between various pythons as I have been very careful to clean everything before I reproduce. Furthermore, moving the installation of tensorflow-metal under python@3.8 from my user package directory to the brew system directory reproduces the error under python@3.8 that was previously working.

Long story short: There is a bug with tensorflow plugin initialization that somehow triggers when tensorflow-metal plugin is installed under /opt/homebrew causing its registration to be called twice. This is avoided when installed in local --user directory.

I have wasted enough time to make this work and I am happy with my theory for now so will drop further looking into it. I just thought I might shed some light for anyone else down the road in case they come across this.

alfaSZ
  • 191
  • 1
  • 4
  • 7
    I am so happy I tried this out. Similar scenario (no conda etc.) and same issue solved with a simple `pip3 uninstall tensorflow-metal` and `pip3 install --user tensorflow-metal`. Thanks – vilfra May 04 '22 at 21:20
1

I have solved the problem after quite some trying.

export OPENBLAS=$(/opt/homebrew/bin/brew --prefix openblas)
export CFLAGS="-falign-functions=8 ${CFLAGS}"

python3 -m venv ~/tensorflow-metal
source ~/tensorflow-metal/bin/activate

python -m pip install -U pip
pip install Cython
pip install --no-use-pep517 numpy==1.19.3

python -m pip install tensorflow-macos
python -m pip install tensorflow-metal

I believe Tensorflow can use both CPU and GPU now.

import tensorflow
tensorflow.config.list_physical_devices()
>>> [PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
Tarık Yılmaz
  • 369
  • 6
  • 10
0

Please try the same after creating a virtual environment.

python3 -m venv ~/tensorflow-metal
source ~/tensorflow-metal/bin/activate
python -m pip install -U pip
python -m pip install tensorflow-macos
python -m pip install tensorflow-metal

Reference- https://developer.apple.com/forums/thread/684889

  • Does not work for Python 3.9. `ERROR: Could not build wheels for numpy which use PEP 517 and cannot be installed directly` and `ERROR: No matching distribution found for h5py~=3.1.0` – Tarık Yılmaz Aug 26 '21 at 12:12
  • Ok. Can you downgrade to python 3.7 /3.8 and check . Ref - https://github.com/numpy/numpy/issues/17569 –  Aug 26 '21 at 13:45
0

I took a more direct approach. The issue is that the tensorflow-plugins uses both sys.path and site.site_packages. The latter repeats one of the paths in sys.path, but through a symlink, so the plugin is loaded twice. So I replaced line 419 of tensorflow/__init__.py:

_site_packages_dirs = list(set(_site_packages_dirs))

by

from pathlib import Path
_site_packages_dirs = {Path(p).resolve() for p in _site_packages_dirs}
_site_packages_dirs = [str(p) for p in _site_packages_dirs]
maparent
  • 31
  • 3