MediaPipe for TouchDesigner

MediaPipe for TouchDesigner

Update Oct 2024
We are using this New version

---

> https://github.com/torinmb/mediapipe-touchdesigner?tab=readme-ov-file

Download the release folder and run the td file inside. When making a new one remember to save it in the same folder as it contains the environment for MediaPipe to run. It will not work outside of it.

On Apple Silicon (M1) using Rosetta2

We will install an emulated x86_64 brew using Rosetta2 alongside native Apple Silicon brew. This way we can install the by TouchDesigner expected python3.7 while also installing the MediaPipe module.

There is a MediaPipe module build for Apple Silicon (M1) machines found at [this Github issue](https://github.com/google/mediapipe/issues/3277), but I couldn't find a native Python 3.7 build for Apple Silicon to run in TouchDesigner.

Install brew x86_64 using Rosetta2

$ arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

Make alias for x86_64 brew

Add this to rc file (.bashrc, .zshrc etc)

alias ibrew="arch -x86_64 /usr/local/bin/brew"

Install python3.7

$ ibrew install python@3.7

Alias for python@3.7 and pip3

Add to rc file

alias iPY37=/usr/local/opt/python@3.7/bin/python3
alias iPIP37=/usr/local/opt/python@3.7/bin/pip3

Now you can use `iPY37` to run x86_64 python@3.7 and `iPIP37` to install packages.

Install Mediapipe for python@3.7 x86_64

$ iPIP37 install mediapipe

Find python@3.7 x86_64 module path

Find the Python Module path of installed x86_64 python@3.7

$ ibrew --prefix

/usr/local

Append `/lib/python3.7/site-packages` to this path to get `/usr/local/lib/python3.7/site-packages`. This is the *Python 64-bit Module Path* you need to fill in TouchDesigner by going to `Preferences -> general -> Python 64-bit Module Path`

Restart TouchDesigner

Mediapipe should be installed! To verify the installation go to `Dialogs -> Textport and DATs`, import the MediaPipe module; `import mediapipe as mp` and list the components; `print(dir(mp.solutions))`

Windows

For windows we need to have a parallel copy of the same Python version on the harddrive. TouchDesigner expects us to work with Python3.7.x so grab this version from the [Python website](https://www.python.org/downloads/windows/). I used [Python 3.7.9](https://www.python.org/ftp/python/3.7.9/python-3.7.9-amd64.exe) for this tutorial.

Install excecutable

Double-click the downloaded executable file and install Python to the `DataStorage` drive `E:/` in a folder named `Python`

Installing MediaPipe

To install MediaPipe for our newly installed Python, open Windows PowerShell and run the following commands

$ python

...
Python 3.7.9
...

$ exit()

• Note: Check if the Python version corresponds with the Python you've installed in step 1.*

Install mediapipe using pip package manager:

$ pip install mediapipe

Open TouchDesigner starter

Download the [MediaPipe starter project](http://interactionstation.wdka.hro.nl/mediawiki/images/a/a5/Mp-starter-toe.zip), unzip it and double-click to open the project in TouchDesigner. If you installed Python3.7 in a different folder than `E:\Python\Lib\site-packages` you need to change the `pythonpath` variable within the `DAT Excecute` OP to the correct folder.

To check if you can use MediaPipe in TouchDesigner, navigate to `Dialogs -> Textport and DATs`, import the MediaPipe module; `import mediapipe as mp` and list the components; `print(dir(mp.solutions))`.

what you also can do is use DAT Execute OP and copy and paste the code below.

Starter code:

# me - this DAT
# 
# frame - the current frame
# state - True if the timeline is paused
# 
# Make sure the corresponding toggle is enabled in the Execute DAT.

def onStart():
	import sys
	
	pythonpath = "E:/Python/Lib/site-packages"
	sys.path = [pythonpath] + sys.path
	return

def onCreate():
	return

def onExit():
	return

def onFrameStart(frame):
	return

def onFrameEnd(frame):
	return

def onPlayStateChange(state):
	return

def onDeviceChange():
	return

def onProjectPreSave():
	return

def onProjectPostSave():
	return

MediaPipe in TouchDesigners

1.faceDetect-CHOP

The code for script CHOP:

first, put a script CHOP on the empty network. and paste the code below:

# me - this DAT
# scriptOp - the OP which is cooking

import numpy as np
import cv2
import sys
import mediapipe as mp

mp_face = mp.solutions.face_detection
face_detection = mp_face.FaceDetection(
    min_detection_confidence=0.7
)

# press 'Setup Parameters' in the OP to call this function to re-create the parameters.
def onSetupParameters(scriptOp):
    page = scriptOp.appendCustomPage('Custom')
    topPar = page.appendTOP('Face', label='Image with face')
    return

# called whenever custom pulse parameter is pushed
def onPulse(par):
	return

def onCook(scriptOp):
    scriptOp.clear()
    topRef = scriptOp.par.Face.eval()
    
    num_faces = 0
    max_area = sys.float_info.min
    width = 0
    height = 0
    xmin = 0.5
    ymin = 0.5
    lx = sys.float_info.max
    ly = sys.float_info.max
    rx = sys.float_info.max
    ry = sys.float_info.max
   
    if topRef:
        img = topRef.numpyArray(delayed=True)
        frame = cv2.cvtColor(img, cv2.COLOR_RGBA2RGB)
        frame *= 255
        frame = frame.astype('uint8')
        results = face_detection.process(frame)
        
        if results.detections:
            num_faces = len(results.detections)
            
            for face in results.detections:
            	area = face.location_data.relative_bounding_box.width * face.location_data.relative_bounding_box.height
            	if area > max_area:
            		width = face.location_data.relative_bounding_box.width
            		height = face.location_data.relative_bounding_box.height
            		xmin = face.location_data.relative_bounding_box.xmin + width/2.0
            		ymin = 1 - (face.location_data.relative_bounding_box.ymin + height/2.0)
            		lx = face.location_data.relative_keypoints[0].x
            		ly = 1 - face.location_data.relative_keypoints[0].y
            		rx = face.location_data.relative_keypoints[1].x
            		ry = 1 - face.location_data.relative_keypoints[1].y
                    
            		max_area = area
        
    tf = scriptOp.appendChan('face')
    tw = scriptOp.appendChan('width')
    th = scriptOp.appendChan('height')
    tx = scriptOp.appendChan('tx')
    ty = scriptOp.appendChan('ty')
    leftx = scriptOp.appendChan('left_eye_x')
    lefty = scriptOp.appendChan('left_eye_y')
    rightx = scriptOp.appendChan('right_eye_x')
    righty = scriptOp.appendChan('right_eye_y')
    nosex = scriptOp.appendChan('nose_x')
    nosey = scriptOp.appendChan('nose_y')
    tf.vals = [num_faces]
    tw.vals = [width]
    th.vals = [height]
    tx.vals = [xmin]
    ty.vals = [ymin]
    
    leftx.vals = [lx]
    lefty.vals = [ly]
    rightx.vals = [rx]
    righty.vals = [ry]
   
   
    scriptOp.rate = me.time.rate

    return

2.Hand Tracking CHOP-the index finger

# me - this DAT
# scriptOp - the OP which is cooking

import numpy as np
import cv2
import mediapipe as mp

mp_hands = mp.solutions.hands
hands = mp_hands.Hands(
	max_num_hands=1,
	min_detection_confidence=0.5,
	min_tracking_confidence=0.5
)

# press 'Setup Parameters' in the OP to call this function to re-create the parameters.
def onSetupParameters(scriptOp):
	page = scriptOp.appendCustomPage('Custom')
	p = page.appendTOP('Image', label='Video image')
	return

# called whenever custom pulse parameter is pushed
def onPulse(par):
	return

def onCook(scriptOp):
	scriptOp.clear()
	input = scriptOp.par.Image.eval().numpyArray(delayed=True)
	image = cv2.cvtColor(input, cv2.COLOR_RGBA2RGB)
	image *= 255
	image = image.astype('uint8')
	results = hands.process(image)

	wrist = []
	index_tip = []
	num_hands = 0
	if results.multi_hand_landmarks:
		num_hands += 1
		for hand in results.multi_hand_landmarks:
			wrist.append(hand.landmark[0])
			index_tip.append(hand.landmark[8])

	tf = scriptOp.appendChan('hands')
	tf.vals = [num_hands]
	
	if len(wrist) > 0:
		twx = scriptOp.appendChan('wrist:x')
		twy = scriptOp.appendChan('wrist:y')

		twx.vals = [wrist[0].x]
		twy.vals = [wrist[0].y]
		
	if len(index_tip) > 0:
		tix = scriptOp.appendChan('index_tip:x')
		tiy = scriptOp.appendChan('index_tip:y')
		
		tix.vals = [index_tip[0].x]
		tiy.vals = [index_tip[0].y]
		
	scriptOp.rate = me.time.rate
	
	return

3.background replacement Selfie segmentation

The TOP Script

# me - this DAT
# scriptOp - the OP which is cooking

import numpy as np
import cv2
import mediapipe as mp

mp_drawing = mp.solutions.drawing_utils
mp_pose = mp.solutions.pose

pose = mp_pose.Pose(
	min_detection_confidence=0.5,
	min_tracking_confidence=0.5,
	enable_segmentation=True
)


# press 'Setup Parameters' in the OP to call this function to re-create the parameters.
def onSetupParameters(scriptOp):
	return

# called whenever custom pulse parameter is pushed
def onPulse(par):
	return


def onCook(scriptOp):
	input = scriptOp.inputs[0].numpyArray(delayed=True)
	if input is not None:
		image = cv2.cvtColor(input, cv2.COLOR_RGBA2RGB)
		image *= 255
		image = image.astype('uint8')
		results = pose.process(image)
		
		if results.segmentation_mask is not None:
			rgb = cv2.cvtColor(results.segmentation_mask, cv2.COLOR_GRAY2RGB)
			rgb = rgb * 255
			rgb = rgb.astype(np.uint8)
			scriptOp.copyNumpyArray(rgb)
		else:
			black = np.zeros(image.shape, dtype=np.uint8)
			scriptOp.copyNumpyArray(black)
	return