Difference between revisions of "Docker"

From Interaction Station Wiki
Jump to navigation Jump to search
(6 intermediate revisions by the same user not shown)
Line 47: Line 47:
 
* Permission Denied on curl and save for docker compose: https://github.com/docker/machine/issues/652
 
* Permission Denied on curl and save for docker compose: https://github.com/docker/machine/issues/652
  
−
=Using Docker with nvidia-docker-composite=
+
=Using Docker with nvidia-docker-compose=
  
 +
*Public docker repository (When doing FROM in Dockerfile, we need to select one of those)
 +
*https://hub.docker.com/
 +
 +
*Dir structure:
 +
*docker-compose.yml
 +
*deepo
 +
*deepo/do_not_finish.sh
 +
*deepo/Dockerfile
 +
*deepo_data (folder that is visible by deepo image)
 +
 +
*docker-compose.yml:
 +
version: '3'
 +
services:
 +
  #machine name
 +
  deepo:
 +
    #container name
 +
    container_name: deepo
 +
    #path to Dockerfile
 +
    build: deepo
 +
    command: sh do_not_finish.sh
 +
    volumes:
 +
      - ./deepo_data:/media/deepo_data
 +
    tty: true
 +
 +
*Dockerfile:
 +
FROM ufoym/deepo
 +
ADD do_not_finish.sh /
 +
*Dockerfiles guide:
 +
*https://rock-it.pl/how-to-write-excellent-dockerfiles/
 +
 +
*do_not_finish.sh:
 +
#!/bin/bash
 +
sh -c 'while :; do sleep 100; done'
 +
 +
*We need that endless loop, because docker-compose closes the container when is deployed
 +
*The endless loop allowed us to use it with a docker exec
 +
 +
==Run it==
 +
*Steps 1 and 2: Within the folder where is the docker-compose.yml file
 +
*sudo nvidia-docker-compose build
 +
*sudo nvidia-docker-compose up
 +
 +
*Step 3: From another terminal:
 +
*sudo nvidia-docker exec -it deepo bash
 +
 +
==Troubleshooting problems==
 
*Check nvidia-docker version (needs to be version 1)
 
*Check nvidia-docker version (needs to be version 1)
 
*nvidia-docker version
 
*nvidia-docker version
Line 55: Line 101:
  
  
−
==Troubleshooting problems==
 
 
*Permission denied: u'./docker-compose.yml
 
*Permission denied: u'./docker-compose.yml
 
*https://github.com/docker/docker-snap/issues/26
 
*https://github.com/docker/docker-snap/issues/26

Revision as of 07:03, 13 October 2018

ML Docker Image installed on the Interaction Station ML computers (Ubuntu 16.04):

Installing Docker CE:

Change Docker root dir using systemd (Don't do this, set volume instead)

Docker - clean up all the volumes

  • sudo docker system prune -a -f --volumes


Installing nvidia-docker:


Installing docker-compose:

Installing nvidia-docker-compose:

Using Docker with nvidia-docker-compose

  • Dir structure:
  • docker-compose.yml
  • deepo
  • deepo/do_not_finish.sh
  • deepo/Dockerfile
  • deepo_data (folder that is visible by deepo image)
  • docker-compose.yml:
version: '3'
services:
  #machine name
  deepo:
    #container name
    container_name: deepo
    #path to Dockerfile
    build: deepo
    command: sh do_not_finish.sh
    volumes:
      - ./deepo_data:/media/deepo_data
    tty: true
  • Dockerfile:

FROM ufoym/deepo ADD do_not_finish.sh /

  • do_not_finish.sh:
  1. !/bin/bash

sh -c 'while :; do sleep 100; done'

  • We need that endless loop, because docker-compose closes the container when is deployed
  • The endless loop allowed us to use it with a docker exec

Run it

  • Steps 1 and 2: Within the folder where is the docker-compose.yml file
  • sudo nvidia-docker-compose build
  • sudo nvidia-docker-compose up
  • Step 3: From another terminal:
  • sudo nvidia-docker exec -it deepo bash

Troubleshooting problems


Deepo

It includes:

  • cudnn
  • theano
  • tensorflow
  • sonnet
  • pytorch
  • keras
  • lasagne
  • mxnet
  • cntk
  • chainer
  • caffe
  • caffe2
  • torch


Installing Deepo:

Run Deepo image with Docker:

  • sudo nvidia-docker run -it ufoym/deepo:gpu bash

Run Deepo image with Docker (with python 2.7):

  • sudo nvidia-docker run -it ufoym/deepo:py27 bash

Setting up ML computers:

  • Linux distribution installed: Ubuntu 16.04

Partition made for machine learning:MachineLearning

  • In Windows: Disk Management -> Resize DataStorage
  • Create new ext4 patition

Mounting the partition automatically:

Get the UUID of the learning:MachineLearning partition

  • sudo blkid

Add partition to fstab:

  • sudo nano /etc/fstab
  • Add at the bottom these two lines:
  • UUID=(id of the MachineLearning partition) /media/MachineLearning rw,suid,dev,auto,user,async,exec 0 2
  • UUID=(id of the DataStorage partition) /media/DataStorage ntfs-3g defaults=en_US.UTF-8 0 0

Give writing permissions to new MachineLearning partition

  • sudo chmod -R a+rwx /media/MachineLearning/
  • Need extra space? Extending the partition

https://askubuntu.com/questions/492054/how-to-extend-my-root-partition

Installing NVIDIA Driver:

  • Set Ubuntu to boot on console mode. Type:
  • sudo apt-get install systemd
  • sudo systemctl set-default multi-user.target
  • sudo reboot now
  • Login and in console mode, type:
  • sudo add-apt-repository ppa:graphics-drivers/ppa
  • sudo apt update
  • sudo apt upgrade
  • For GeForce 1070Ti (07/2018), type:
  • sudo apt-get install nvidia-390
  • Re-set Ubuntu to boot on graphical mode. Type:
  • sudo systemctl set-default graphical.target
  • sudo reboot now

Checking if Nvidia Driver is properly installed. Type:

  • nvidia-smi
  • nvidia-settings

Installing CUDA 9.0 for Ubuntu 16.04 (the latest version is not supported by TensorFlow):

Checking if CUDA is properly installed. Type:

  • nvcc --version

Resources used:


Other options:

NTFS fstab wizard:

  • sudo apt-get install ntfs-config
  • sudo ntfs-config

Format large capacity HD with fs ExFat for having access to it from Ubuntu:

  • On Windows 10
  • cmd
  • diskpart
  • select disk '#' (where # is the number of the target drive)
  • list part
  • select part # (where # is the number of the partition)
  • format fs=exfat QUICK
Retrieved from "http://interactionstation.wdka.hro.nl/mediawiki/index.php?title=Docker&oldid=3249"