MediaPipe for TouchDesigner

MediaPipe for TouchDesigner

On Apple Silicon (M1) using Rosetta2

We will install an emulated x86_64 brew using Rosetta2 alongside native Apple Silicon brew. This way we can install the by TouchDesigner expected python3.7 while also installing the MediaPipe module.

There is a MediaPipe module build for Apple Silicon (M1) machines found at [this Github issue](https://github.com/google/mediapipe/issues/3277), but I couldn't find a native Python 3.7 build for Apple Silicon to run in TouchDesigner.

Install brew x86_64 using Rosetta2

$ arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

Make alias for x86_64 brew

Add this to rc file (.bashrc, .zshrc etc)

alias ibrew="arch -x86_64 /usr/local/bin/brew"

Install python3.7

$ ibrew install python@3.7

Alias for python@3.7 and pip3

Add to rc file

alias iPY37=/usr/local/opt/python@3.7/bin/python3
alias iPIP37=/usr/local/opt/python@3.7/bin/pip3

Now you can use `iPY37` to run x86_64 python@3.7 and `iPIP37` to install packages.

Install Mediapipe for python@3.7 x86_64

$ iPIP37 install mediapipe

Find python@3.7 x86_64 module path

Find the Python Module path of installed x86_64 python@3.7

$ ibrew --prefix

/usr/local

Append `/lib/python3.7/site-packages` to this path to get `/usr/local/lib/python3.7/site-packages`. This is the *Python 64-bit Module Path* you need to fill in TouchDesigner by going to `Preferences -> general -> Python 64-bit Module Path`

Restart TouchDesigner

Mediapipe should be installed! To verify the installation go to `Dialogs -> Textport and DATs`, import the MediaPipe module; `import mediapipe as mp` and list the components; `print(dir(mp.solutions))`

Windows

For windows we need to have a parallel copy of the same Python version on the harddrive. TouchDesigner expects us to work with Python3.7.x so grab this version from the [Python website](https://www.python.org/downloads/windows/). I used [Python 3.7.9](https://www.python.org/ftp/python/3.7.9/python-3.7.9-amd64.exe) for this tutorial.

Install excecutable

Double-click the downloaded executable file and install Python to the `DataStorage` drive `E:/` in a folder named `Python`

Installing MediaPipe

To install MediaPipe for our newly installed Python, open Windows PowerShell and run the following commands

$ python

...
Python 3.7.9
...

$ exit()

• Note: Check if the Python version corresponds with the Python you've installed in step 1.*

Install mediapipe using pip package manager:

$ pip install mediapipe

Open TouchDesigner starter

Download the [MediaPipe starter project](http://interactionstation.wdka.hro.nl/mediawiki/images/a/a5/Mp-starter-toe.zip), unzip it and double-click to open the project in TouchDesigner. If you installed Python3.7 in a different folder than `E:\Python\Lib\site-packages` you need to change the `pythonpath` variable within the `DAT Excecute` OP to the correct folder.

To check if you can use MediaPipe in TouchDesigner, navigate to `Dialogs -> Textport and DATs`, import the MediaPipe module; `import mediapipe as mp` and list the components; `print(dir(mp.solutions))`.

what you also can do is use DAT Execute OP and copy and paste the code below.

Starter code:

 1# me - this DAT
 2# 
 3# frame - the current frame
 4# state - True if the timeline is paused
 5# 
 6# Make sure the corresponding toggle is enabled in the Execute DAT.
 7
 8def onStart():
 9	import sys
10	
11	pythonpath = "E:/Python/Lib/site-packages"
12	sys.path = [pythonpath] + sys.path
13	return
14
15def onCreate():
16	return
17
18def onExit():
19	return
20
21def onFrameStart(frame):
22	return
23
24def onFrameEnd(frame):
25	return
26
27def onPlayStateChange(state):
28	return
29
30def onDeviceChange():
31	return
32
33def onProjectPreSave():
34	return
35
36def onProjectPostSave():
37	return

MediaPipe examples

download link:

https://we.tl/t-RihN6L38gw

faceDetect-CHOP

1.http://interactionstation.wdka.hro.nl/mediawiki/images/c/c5/Butter-FaceDetectionScriptCHOP.toe.zip

2.http://interactionstation.wdka.hro.nl/mediawiki/images/a/af/Cloud-face-voice-FaceDetectionScriptCHOP.20.toe.zip

FaceMeshSOP

1.http://interactionstation.wdka.hro.nl/mediawiki/images/1/11/MediaPipeFaceMeshSOP.zip

Hand-CHOP-index-finger

1.http://interactionstation.wdka.hro.nl/mediawiki/images/f/f2/Hand-index.zip

Hand-CHOP-all-fingers

1. http://interactionstation.wdka.hro.nl/mediawiki/images/d/de/MediaPipeHandCHOP-all-fingers.toe.zip

PoseCHOP

1.http://interactionstation.wdka.hro.nl/mediawiki/images/2/2d/MediaPipePoseCHOP.toe.zip

SegmentationMask

1. http://interactionstation.wdka.hro.nl/mediawiki/images/7/7e/SegmentationMask.zip