Difference between revisions of "MediaPipe for TouchDesigner"

From Interaction Station Wiki
Jump to navigation Jump to search
(Created page with "# MediaPipe for TouchDesigner ### On Apple Silicon (M1) using Rosetta2 We will install an emulated x86_64 brew using Rosetta2 alongside native Apple Silicon brew. This way w...")
 
 
(29 intermediate revisions by 3 users not shown)
Line 1: Line 1:
−
# MediaPipe for TouchDesigner
+
== MediaPipe for TouchDesigner ==
  
−
### On Apple Silicon (M1) using Rosetta2
+
<div style="color:red;font-family:monospace;background-color:lightyellow;">
 +
Update Oct 2024 <br>
 +
We are using this New version
 +
----> https://github.com/torinmb/mediapipe-touchdesigner?tab=readme-ov-file
 +
Download the release folder and run the td file inside. When making a new one remember to save it in the same folder as it contains the environment for MediaPipe to run. It will not work outside of it.
 +
</div>
 +
 
 +
 
 +
=== On Apple Silicon (M1) using Rosetta2 ===
  
 
We will install an emulated x86_64 brew using Rosetta2 alongside native Apple Silicon brew. This way we can install the by TouchDesigner expected python3.7 while also installing the MediaPipe module.
 
We will install an emulated x86_64 brew using Rosetta2 alongside native Apple Silicon brew. This way we can install the by TouchDesigner expected python3.7 while also installing the MediaPipe module.
Line 7: Line 15:
 
There is a MediaPipe module build for Apple Silicon (M1) machines found at [this Github issue](https://github.com/google/mediapipe/issues/3277), but I couldn't find a native Python 3.7 build for Apple Silicon to run in TouchDesigner.
 
There is a MediaPipe module build for Apple Silicon (M1) machines found at [this Github issue](https://github.com/google/mediapipe/issues/3277), but I couldn't find a native Python 3.7 build for Apple Silicon to run in TouchDesigner.
  
 +
=== Install brew x86_64 using Rosetta2 ===
  
−
##### 1. Install brew x86_64 using Rosetta2
+
<syntaxhighlight lang=c style="border:1px dashed black; max-width: 950px;">
−
 
 
−
```bash
 
 
$ arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
 
$ arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
−
```
+
</syntaxhighlight>
  
−
##### 2. Make alias for x86_64 brew
+
=== Make alias for x86_64 brew ===
  
 
Add this to rc file (.bashrc, .zshrc etc)
 
Add this to rc file (.bashrc, .zshrc etc)
  
−
```bash
+
<syntaxhighlight lang=c style="border:1px dashed black; max-width: 950px;">
 
alias ibrew="arch -x86_64 /usr/local/bin/brew"
 
alias ibrew="arch -x86_64 /usr/local/bin/brew"
−
```
+
</syntaxhighlight>
  
−
##### 3. Install python3.7
+
=== Install python3.7 ===
  
−
```bash
+
<syntaxhighlight lang=c style="border:1px dashed black; max-width: 950px;">
 
$ ibrew install python@3.7
 
$ ibrew install python@3.7
−
```
+
</syntaxhighlight>
−
 
+
−
##### 4. Alias for python@3.7 and pip3
+
=== Alias for python@3.7 and pip3 ===
  
 
Add to rc file
 
Add to rc file
  
−
```bash
+
<syntaxhighlight lang=c style="border:1px dashed black; max-width: 950px;">
 
alias iPY37=/usr/local/opt/python@3.7/bin/python3
 
alias iPY37=/usr/local/opt/python@3.7/bin/python3
 
alias iPIP37=/usr/local/opt/python@3.7/bin/pip3
 
alias iPIP37=/usr/local/opt/python@3.7/bin/pip3
−
```
+
</syntaxhighlight>
  
 
Now you can use `iPY37` to run x86_64 python@3.7 and `iPIP37` to install packages.
 
Now you can use `iPY37` to run x86_64 python@3.7 and `iPIP37` to install packages.
  
−
##### 5. Install Mediapipe for python@3.7 x86_64
+
=== Install Mediapipe for python@3.7 x86_64 ===
  
−
```bash
+
<syntaxhighlight lang=c style="border:1px dashed black; max-width: 950px;">
 
$ iPIP37 install mediapipe
 
$ iPIP37 install mediapipe
−
```
+
</syntaxhighlight>
  
−
##### 6. Find python@3.7 x86_64 module path
+
=== Find python@3.7 x86_64 module path ===
  
 
Find the Python Module path of installed x86_64 python@3.7
 
Find the Python Module path of installed x86_64 python@3.7
  
−
```
+
<syntaxhighlight lang=c style="border:1px dashed black; max-width: 950px;">
 
$ ibrew --prefix
 
$ ibrew --prefix
  
 
/usr/local
 
/usr/local
−
```
+
</syntaxhighlight>
  
 
Append `/lib/python3.7/site-packages` to this path to get `/usr/local/lib/python3.7/site-packages`. This is the *Python 64-bit Module Path* you need to fill in TouchDesigner by going to `Preferences -> general -> Python 64-bit Module Path`
 
Append `/lib/python3.7/site-packages` to this path to get `/usr/local/lib/python3.7/site-packages`. This is the *Python 64-bit Module Path* you need to fill in TouchDesigner by going to `Preferences -> general -> Python 64-bit Module Path`
  
−
##### 7. Restart TouchDesigner
+
=== Restart TouchDesigner ===
  
 
Mediapipe should be installed! To verify the installation go to `Dialogs -> Textport and DATs`, import the MediaPipe module; `import mediapipe as mp` and list the components; `print(dir(mp.solutions))`
 
Mediapipe should be installed! To verify the installation go to `Dialogs -> Textport and DATs`, import the MediaPipe module; `import mediapipe as mp` and list the components; `print(dir(mp.solutions))`
  
−
### Windows
+
== Windows ==
  
 
For windows we need to have a parallel copy of the same Python version on the harddrive. TouchDesigner expects us to work with Python3.7.x so grab this version from the [Python website](https://www.python.org/downloads/windows/). I used [Python 3.7.9](https://www.python.org/ftp/python/3.7.9/python-3.7.9-amd64.exe) for this tutorial.
 
For windows we need to have a parallel copy of the same Python version on the harddrive. TouchDesigner expects us to work with Python3.7.x so grab this version from the [Python website](https://www.python.org/downloads/windows/). I used [Python 3.7.9](https://www.python.org/ftp/python/3.7.9/python-3.7.9-amd64.exe) for this tutorial.
  
−
##### 1. Install excecutable
+
=== Install excecutable ===
  
 
Double-click the downloaded executable file and install Python to the `DataStorage` drive `E:/` in a folder named `Python`
 
Double-click the downloaded executable file and install Python to the `DataStorage` drive `E:/` in a folder named `Python`
  
−
##### 2. Installing MediaPipe
+
=== Installing MediaPipe ===
  
 
To install MediaPipe for our newly installed Python, open Windows PowerShell and run the following commands
 
To install MediaPipe for our newly installed Python, open Windows PowerShell and run the following commands
  
−
```dos
+
<syntaxhighlight lang=c style="border:1px dashed black; max-width: 950px;">
 
$ python
 
$ python
  
Line 81: Line 88:
  
 
$ exit()
 
$ exit()
−
```
+
</syntaxhighlight>
  
 
*Note: Check if the Python version corresponds with the Python you've installed in step 1.*
 
*Note: Check if the Python version corresponds with the Python you've installed in step 1.*
Line 87: Line 94:
 
Install mediapipe using pip package manager:
 
Install mediapipe using pip package manager:
  
−
```
+
<syntaxhighlight lang=c style="border:1px dashed black; max-width: 950px;">
 
$ pip install mediapipe
 
$ pip install mediapipe
−
```
+
</syntaxhighlight>
  
−
##### 3. Open TouchDesigner starter
+
=== Open TouchDesigner starter ===
  
 
Download the [MediaPipe starter project](http://interactionstation.wdka.hro.nl/mediawiki/images/a/a5/Mp-starter-toe.zip), unzip it and double-click to open the project in TouchDesigner. If you installed Python3.7 in a different folder than `E:\Python\Lib\site-packages` you need to change the `pythonpath` variable within the `DAT Excecute` OP to the correct folder.
 
Download the [MediaPipe starter project](http://interactionstation.wdka.hro.nl/mediawiki/images/a/a5/Mp-starter-toe.zip), unzip it and double-click to open the project in TouchDesigner. If you installed Python3.7 in a different folder than `E:\Python\Lib\site-packages` you need to change the `pythonpath` variable within the `DAT Excecute` OP to the correct folder.
  
 
To check if you can use MediaPipe in TouchDesigner, navigate to `Dialogs -> Textport and DATs`, import the MediaPipe module; `import mediapipe as mp` and list the components; `print(dir(mp.solutions))`.
 
To check if you can use MediaPipe in TouchDesigner, navigate to `Dialogs -> Textport and DATs`, import the MediaPipe module; `import mediapipe as mp` and list the components; `print(dir(mp.solutions))`.
 +
 +
 +
'''what you also can do is use DAT Execute OP and copy and paste the code below.'''
 +
 +
'''Starter code:'''
 +
 +
<pre>
 +
# me - this DAT
 +
#
 +
# frame - the current frame
 +
# state - True if the timeline is paused
 +
#
 +
# Make sure the corresponding toggle is enabled in the Execute DAT.
 +
 +
def onStart():
 +
import sys
 +
 +
pythonpath = "E:/Python/Lib/site-packages"
 +
sys.path = [pythonpath] + sys.path
 +
return
 +
 +
def onCreate():
 +
return
 +
 +
def onExit():
 +
return
 +
 +
def onFrameStart(frame):
 +
return
 +
 +
def onFrameEnd(frame):
 +
return
 +
 +
def onPlayStateChange(state):
 +
return
 +
 +
def onDeviceChange():
 +
return
 +
 +
def onProjectPreSave():
 +
return
 +
 +
def onProjectPostSave():
 +
return
 +
 +
</pre>
 +
 +
== MediaPipe in TouchDesigners==
 +
 +
 +
 +
===1.faceDetect-CHOP===
 +
 +
The code for script CHOP:
 +
 +
first, put a script CHOP on the empty network. and paste the code below:
 +
 +
 +
<pre>
 +
# me - this DAT
 +
# scriptOp - the OP which is cooking
 +
 +
import numpy as np
 +
import cv2
 +
import sys
 +
import mediapipe as mp
 +
 +
mp_face = mp.solutions.face_detection
 +
face_detection = mp_face.FaceDetection(
 +
    min_detection_confidence=0.7
 +
)
 +
 +
# press 'Setup Parameters' in the OP to call this function to re-create the parameters.
 +
def onSetupParameters(scriptOp):
 +
    page = scriptOp.appendCustomPage('Custom')
 +
    topPar = page.appendTOP('Face', label='Image with face')
 +
    return
 +
 +
# called whenever custom pulse parameter is pushed
 +
def onPulse(par):
 +
return
 +
 +
def onCook(scriptOp):
 +
    scriptOp.clear()
 +
    topRef = scriptOp.par.Face.eval()
 +
   
 +
    num_faces = 0
 +
    max_area = sys.float_info.min
 +
    width = 0
 +
    height = 0
 +
    xmin = 0.5
 +
    ymin = 0.5
 +
    lx = sys.float_info.max
 +
    ly = sys.float_info.max
 +
    rx = sys.float_info.max
 +
    ry = sys.float_info.max
 +
 
 +
    if topRef:
 +
        img = topRef.numpyArray(delayed=True)
 +
        frame = cv2.cvtColor(img, cv2.COLOR_RGBA2RGB)
 +
        frame *= 255
 +
        frame = frame.astype('uint8')
 +
        results = face_detection.process(frame)
 +
       
 +
        if results.detections:
 +
            num_faces = len(results.detections)
 +
           
 +
            for face in results.detections:
 +
            area = face.location_data.relative_bounding_box.width * face.location_data.relative_bounding_box.height
 +
            if area > max_area:
 +
            width = face.location_data.relative_bounding_box.width
 +
            height = face.location_data.relative_bounding_box.height
 +
            xmin = face.location_data.relative_bounding_box.xmin + width/2.0
 +
            ymin = 1 - (face.location_data.relative_bounding_box.ymin + height/2.0)
 +
            lx = face.location_data.relative_keypoints[0].x
 +
            ly = 1 - face.location_data.relative_keypoints[0].y
 +
            rx = face.location_data.relative_keypoints[1].x
 +
            ry = 1 - face.location_data.relative_keypoints[1].y
 +
                   
 +
            max_area = area
 +
       
 +
    tf = scriptOp.appendChan('face')
 +
    tw = scriptOp.appendChan('width')
 +
    th = scriptOp.appendChan('height')
 +
    tx = scriptOp.appendChan('tx')
 +
    ty = scriptOp.appendChan('ty')
 +
    leftx = scriptOp.appendChan('left_eye_x')
 +
    lefty = scriptOp.appendChan('left_eye_y')
 +
    rightx = scriptOp.appendChan('right_eye_x')
 +
    righty = scriptOp.appendChan('right_eye_y')
 +
    nosex = scriptOp.appendChan('nose_x')
 +
    nosey = scriptOp.appendChan('nose_y')
 +
    tf.vals = [num_faces]
 +
    tw.vals = [width]
 +
    th.vals = [height]
 +
    tx.vals = [xmin]
 +
    ty.vals = [ymin]
 +
   
 +
    leftx.vals = [lx]
 +
    lefty.vals = [ly]
 +
    rightx.vals = [rx]
 +
    righty.vals = [ry]
 +
 
 +
 
 +
    scriptOp.rate = me.time.rate
 +
 +
    return
 +
</pre>
 +
 +
===2.Hand Tracking CHOP-the index finger===
 +
 +
<pre>
 +
# me - this DAT
 +
# scriptOp - the OP which is cooking
 +
 +
import numpy as np
 +
import cv2
 +
import mediapipe as mp
 +
 +
mp_hands = mp.solutions.hands
 +
hands = mp_hands.Hands(
 +
max_num_hands=1,
 +
min_detection_confidence=0.5,
 +
min_tracking_confidence=0.5
 +
)
 +
 +
# press 'Setup Parameters' in the OP to call this function to re-create the parameters.
 +
def onSetupParameters(scriptOp):
 +
page = scriptOp.appendCustomPage('Custom')
 +
p = page.appendTOP('Image', label='Video image')
 +
return
 +
 +
# called whenever custom pulse parameter is pushed
 +
def onPulse(par):
 +
return
 +
 +
def onCook(scriptOp):
 +
scriptOp.clear()
 +
input = scriptOp.par.Image.eval().numpyArray(delayed=True)
 +
image = cv2.cvtColor(input, cv2.COLOR_RGBA2RGB)
 +
image *= 255
 +
image = image.astype('uint8')
 +
results = hands.process(image)
 +
 +
wrist = []
 +
index_tip = []
 +
num_hands = 0
 +
if results.multi_hand_landmarks:
 +
num_hands += 1
 +
for hand in results.multi_hand_landmarks:
 +
wrist.append(hand.landmark[0])
 +
index_tip.append(hand.landmark[8])
 +
 +
tf = scriptOp.appendChan('hands')
 +
tf.vals = [num_hands]
 +
 +
if len(wrist) > 0:
 +
twx = scriptOp.appendChan('wrist:x')
 +
twy = scriptOp.appendChan('wrist:y')
 +
 +
twx.vals = [wrist[0].x]
 +
twy.vals = [wrist[0].y]
 +
 +
if len(index_tip) > 0:
 +
tix = scriptOp.appendChan('index_tip:x')
 +
tiy = scriptOp.appendChan('index_tip:y')
 +
 +
tix.vals = [index_tip[0].x]
 +
tiy.vals = [index_tip[0].y]
 +
 +
scriptOp.rate = me.time.rate
 +
 +
return
 +
</pre>
 +
 +
===2.Hand Tracking CHOP-all fingers===
 +
 +
<syntaxhighlight lang="python" line='line'>
 +
 +
</syntaxhighlight>
 +
===3.background replacement Selfie segmentation ===
 +
 +
'''The TOP Script'''
 +
 +
 +
<pre>
 +
# me - this DAT
 +
# scriptOp - the OP which is cooking
 +
 +
import numpy as np
 +
import cv2
 +
import mediapipe as mp
 +
 +
mp_drawing = mp.solutions.drawing_utils
 +
mp_pose = mp.solutions.pose
 +
 +
pose = mp_pose.Pose(
 +
min_detection_confidence=0.5,
 +
min_tracking_confidence=0.5,
 +
enable_segmentation=True
 +
)
 +
 +
 +
# press 'Setup Parameters' in the OP to call this function to re-create the parameters.
 +
def onSetupParameters(scriptOp):
 +
return
 +
 +
# called whenever custom pulse parameter is pushed
 +
def onPulse(par):
 +
return
 +
 +
 +
def onCook(scriptOp):
 +
input = scriptOp.inputs[0].numpyArray(delayed=True)
 +
if input is not None:
 +
image = cv2.cvtColor(input, cv2.COLOR_RGBA2RGB)
 +
image *= 255
 +
image = image.astype('uint8')
 +
results = pose.process(image)
 +
 +
if results.segmentation_mask is not None:
 +
rgb = cv2.cvtColor(results.segmentation_mask, cv2.COLOR_GRAY2RGB)
 +
rgb = rgb * 255
 +
rgb = rgb.astype(np.uint8)
 +
scriptOp.copyNumpyArray(rgb)
 +
else:
 +
black = np.zeros(image.shape, dtype=np.uint8)
 +
scriptOp.copyNumpyArray(black)
 +
return
 +
 +
</pre>
 +
 +
===4.posture tracking PoseCHOP ===
 +
 +
<syntaxhighlight lang="python" line='line'>
 +
 +
</syntaxhighlight>
 +
 +
==[[examples link:]]==
 +
 +
[[Category:TouchDesigner]]

Latest revision as of 15:15, 1 October 2024

MediaPipe for TouchDesigner

Update Oct 2024
We are using this New version

> https://github.com/torinmb/mediapipe-touchdesigner?tab=readme-ov-file

Download the release folder and run the td file inside. When making a new one remember to save it in the same folder as it contains the environment for MediaPipe to run. It will not work outside of it.


On Apple Silicon (M1) using Rosetta2

We will install an emulated x86_64 brew using Rosetta2 alongside native Apple Silicon brew. This way we can install the by TouchDesigner expected python3.7 while also installing the MediaPipe module.

There is a MediaPipe module build for Apple Silicon (M1) machines found at [this Github issue](https://github.com/google/mediapipe/issues/3277), but I couldn't find a native Python 3.7 build for Apple Silicon to run in TouchDesigner.

Install brew x86_64 using Rosetta2

$ arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

Make alias for x86_64 brew

Add this to rc file (.bashrc, .zshrc etc) 

alias ibrew="arch -x86_64 /usr/local/bin/brew"

Install python3.7

$ ibrew install python@3.7

Alias for python@3.7 and pip3

Add to rc file 

alias iPY37=/usr/local/opt/python@3.7/bin/python3
alias iPIP37=/usr/local/opt/python@3.7/bin/pip3

Now you can use `iPY37` to run x86_64 python@3.7 and `iPIP37` to install packages.

Install Mediapipe for python@3.7 x86_64

$ iPIP37 install mediapipe

Find python@3.7 x86_64 module path

Find the Python Module path of installed x86_64 python@3.7 

$ ibrew --prefix

/usr/local

Append `/lib/python3.7/site-packages` to this path to get `/usr/local/lib/python3.7/site-packages`. This is the *Python 64-bit Module Path* you need to fill in TouchDesigner by going to `Preferences -> general -> Python 64-bit Module Path`

Restart TouchDesigner

Mediapipe should be installed! To verify the installation go to `Dialogs -> Textport and DATs`, import the MediaPipe module; `import mediapipe as mp` and list the components; `print(dir(mp.solutions))`

Windows

For windows we need to have a parallel copy of the same Python version on the harddrive. TouchDesigner expects us to work with Python3.7.x so grab this version from the [Python website](https://www.python.org/downloads/windows/). I used [Python 3.7.9](https://www.python.org/ftp/python/3.7.9/python-3.7.9-amd64.exe) for this tutorial.

Install excecutable

Double-click the downloaded executable file and install Python to the `DataStorage` drive `E:/` in a folder named `Python`

Installing MediaPipe

To install MediaPipe for our newly installed Python, open Windows PowerShell and run the following commands 

$ python

...
Python 3.7.9
...

$ exit()
  • Note: Check if the Python version corresponds with the Python you've installed in step 1.*

Install mediapipe using pip package manager: 

$ pip install mediapipe

Open TouchDesigner starter

Download the [MediaPipe starter project](http://interactionstation.wdka.hro.nl/mediawiki/images/a/a5/Mp-starter-toe.zip), unzip it and double-click to open the project in TouchDesigner. If you installed Python3.7 in a different folder than `E:\Python\Lib\site-packages` you need to change the `pythonpath` variable within the `DAT Excecute` OP to the correct folder.

To check if you can use MediaPipe in TouchDesigner, navigate to `Dialogs -> Textport and DATs`, import the MediaPipe module; `import mediapipe as mp` and list the components; `print(dir(mp.solutions))`. 


what you also can do is use DAT Execute OP and copy and paste the code below.

Starter code: 

# me - this DAT
# 
# frame - the current frame
# state - True if the timeline is paused
# 
# Make sure the corresponding toggle is enabled in the Execute DAT.

def onStart():
	import sys
	
	pythonpath = "E:/Python/Lib/site-packages"
	sys.path = [pythonpath] + sys.path
	return

def onCreate():
	return

def onExit():
	return

def onFrameStart(frame):
	return

def onFrameEnd(frame):
	return

def onPlayStateChange(state):
	return

def onDeviceChange():
	return

def onProjectPreSave():
	return

def onProjectPostSave():
	return

MediaPipe in TouchDesigners

1.faceDetect-CHOP

The code for script CHOP:

first, put a script CHOP on the empty network. and paste the code below: 


# me - this DAT
# scriptOp - the OP which is cooking

import numpy as np
import cv2
import sys
import mediapipe as mp

mp_face = mp.solutions.face_detection
face_detection = mp_face.FaceDetection(
    min_detection_confidence=0.7
)

# press 'Setup Parameters' in the OP to call this function to re-create the parameters.
def onSetupParameters(scriptOp):
    page = scriptOp.appendCustomPage('Custom')
    topPar = page.appendTOP('Face', label='Image with face')
    return

# called whenever custom pulse parameter is pushed
def onPulse(par):
	return

def onCook(scriptOp):
    scriptOp.clear()
    topRef = scriptOp.par.Face.eval()
    
    num_faces = 0
    max_area = sys.float_info.min
    width = 0
    height = 0
    xmin = 0.5
    ymin = 0.5
    lx = sys.float_info.max
    ly = sys.float_info.max
    rx = sys.float_info.max
    ry = sys.float_info.max
   
    if topRef:
        img = topRef.numpyArray(delayed=True)
        frame = cv2.cvtColor(img, cv2.COLOR_RGBA2RGB)
        frame *= 255
        frame = frame.astype('uint8')
        results = face_detection.process(frame)
        
        if results.detections:
            num_faces = len(results.detections)
            
            for face in results.detections:
            	area = face.location_data.relative_bounding_box.width * face.location_data.relative_bounding_box.height
            	if area > max_area:
            		width = face.location_data.relative_bounding_box.width
            		height = face.location_data.relative_bounding_box.height
            		xmin = face.location_data.relative_bounding_box.xmin + width/2.0
            		ymin = 1 - (face.location_data.relative_bounding_box.ymin + height/2.0)
            		lx = face.location_data.relative_keypoints[0].x
            		ly = 1 - face.location_data.relative_keypoints[0].y
            		rx = face.location_data.relative_keypoints[1].x
            		ry = 1 - face.location_data.relative_keypoints[1].y
                    
            		max_area = area
        
    tf = scriptOp.appendChan('face')
    tw = scriptOp.appendChan('width')
    th = scriptOp.appendChan('height')
    tx = scriptOp.appendChan('tx')
    ty = scriptOp.appendChan('ty')
    leftx = scriptOp.appendChan('left_eye_x')
    lefty = scriptOp.appendChan('left_eye_y')
    rightx = scriptOp.appendChan('right_eye_x')
    righty = scriptOp.appendChan('right_eye_y')
    nosex = scriptOp.appendChan('nose_x')
    nosey = scriptOp.appendChan('nose_y')
    tf.vals = [num_faces]
    tw.vals = [width]
    th.vals = [height]
    tx.vals = [xmin]
    ty.vals = [ymin]
    
    leftx.vals = [lx]
    lefty.vals = [ly]
    rightx.vals = [rx]
    righty.vals = [ry]
   
   
    scriptOp.rate = me.time.rate

    return

2.Hand Tracking CHOP-the index finger

# me - this DAT
# scriptOp - the OP which is cooking

import numpy as np
import cv2
import mediapipe as mp

mp_hands = mp.solutions.hands
hands = mp_hands.Hands(
	max_num_hands=1,
	min_detection_confidence=0.5,
	min_tracking_confidence=0.5
)

# press 'Setup Parameters' in the OP to call this function to re-create the parameters.
def onSetupParameters(scriptOp):
	page = scriptOp.appendCustomPage('Custom')
	p = page.appendTOP('Image', label='Video image')
	return

# called whenever custom pulse parameter is pushed
def onPulse(par):
	return

def onCook(scriptOp):
	scriptOp.clear()
	input = scriptOp.par.Image.eval().numpyArray(delayed=True)
	image = cv2.cvtColor(input, cv2.COLOR_RGBA2RGB)
	image *= 255
	image = image.astype('uint8')
	results = hands.process(image)

	wrist = []
	index_tip = []
	num_hands = 0
	if results.multi_hand_landmarks:
		num_hands += 1
		for hand in results.multi_hand_landmarks:
			wrist.append(hand.landmark[0])
			index_tip.append(hand.landmark[8])

	tf = scriptOp.appendChan('hands')
	tf.vals = [num_hands]
	
	if len(wrist) > 0:
		twx = scriptOp.appendChan('wrist:x')
		twy = scriptOp.appendChan('wrist:y')

		twx.vals = [wrist[0].x]
		twy.vals = [wrist[0].y]
		
	if len(index_tip) > 0:
		tix = scriptOp.appendChan('index_tip:x')
		tiy = scriptOp.appendChan('index_tip:y')
		
		tix.vals = [index_tip[0].x]
		tiy.vals = [index_tip[0].y]
		
	scriptOp.rate = me.time.rate
	
	return

2.Hand Tracking CHOP-all fingers

3.background replacement Selfie segmentation

The TOP Script 


# me - this DAT
# scriptOp - the OP which is cooking

import numpy as np
import cv2
import mediapipe as mp

mp_drawing = mp.solutions.drawing_utils
mp_pose = mp.solutions.pose

pose = mp_pose.Pose(
	min_detection_confidence=0.5,
	min_tracking_confidence=0.5,
	enable_segmentation=True
)


# press 'Setup Parameters' in the OP to call this function to re-create the parameters.
def onSetupParameters(scriptOp):
	return

# called whenever custom pulse parameter is pushed
def onPulse(par):
	return


def onCook(scriptOp):
	input = scriptOp.inputs[0].numpyArray(delayed=True)
	if input is not None:
		image = cv2.cvtColor(input, cv2.COLOR_RGBA2RGB)
		image *= 255
		image = image.astype('uint8')
		results = pose.process(image)
		
		if results.segmentation_mask is not None:
			rgb = cv2.cvtColor(results.segmentation_mask, cv2.COLOR_GRAY2RGB)
			rgb = rgb * 255
			rgb = rgb.astype(np.uint8)
			scriptOp.copyNumpyArray(rgb)
		else:
			black = np.zeros(image.shape, dtype=np.uint8)
			scriptOp.copyNumpyArray(black)
	return

4.posture tracking PoseCHOP

examples link:

Retrieved from "http://interactionstation.wdka.hro.nl/mediawiki/index.php?title=MediaPipe_for_TouchDesigner&oldid=11563"