OpenCV with Raspberry Pi Camera Face Detection Tutorial - Robotics with Python Raspberry Pi and GoPiGo p.7

Next, we're going to touch on using OpenCV with the Raspberry Pi's camera, giving our robot the gift of sight. There are many steps involved to this process, so there's a lot that is about to be thrown your way. If at any point you're stuck/lost/whatever, feel free to ask questions on the video and I will try to help where possible. There are a lot of moving parts here. If all else fails, I have hosted my Raspberry Pi image: OpenCV and GoPiGo Tutorials Image. It's a 16GB image, and, since reformatting may leave you with too-little space, you may very likely need a larger than 16GB SD card (so, 32GB). Here's a $9 32GB Micro SD card with the adapter. Again, you want to make sure you have a micro-sd with the adapter for two major reasons. Most SD card readers on computers, which you will use to mount the image, are for full-size SD cards, but the Raspberry Pi Model B+ and onward take only a micro SD card, so you need both!

First, we want to get OpenCV. OpenCV stands for Open Computer Vision, and it is an open source computer vision and machine learning library. To start, you will need to get OpenCV on to your Raspberry Pi. I used the following: http://mitchtech.net/raspberry-pi-opencv/

Keep in mind, the "make" part of this tutorial will take 9-10 hours on a Raspberry Pi Model B+. The Raspberry Pi 2 will do it in more like 2-4 hours. Either way, it will take a while. I just did it overnight one night.

In case you are having any trouble getting OpenCV set up on your SD card, I have hosted an image of my SD card with everything up to this point, including OpenCV. It is an image from a 16gb SD card, which means you very well might need a 32gb SD card, since you might not have enough space after formatting.

Once you've done that, you're ready to interact with OpenCV. The following is an example of code I found here

import io
import picamera
import cv2
import numpy

#Create a memory stream so photos doesn't need to be saved in a file
stream = io.BytesIO()

#Get the picture (low resolution, so it should be quite fast)
#Here you can also specify other parameters (e.g.:rotate the image)
with picamera.PiCamera() as camera:
    camera.resolution = (320, 240)
    camera.capture(stream, format='jpeg')

#Convert the picture into a numpy array
buff = numpy.fromstring(stream.getvalue(), dtype=numpy.uint8)

#Now creates an OpenCV image
image = cv2.imdecode(buff, 1)

#Load a cascade file for detecting faces
face_cascade = cv2.CascadeClassifier('/usr/share/opencv/haarcascades/haarcascade_frontalface_alt.xml')

#Convert to grayscale
gray = cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)

#Look for faces in the image using the loaded cascade file
faces = face_cascade.detectMultiScale(gray, 1.1, 5)

print "Found "+str(len(faces))+" face(s)"

#Draw a rectangle around every found face
for (x,y,w,h) in faces:
    cv2.rectangle(image,(x,y),(x+w,y+h),(255,255,0),2)

#Save the result image
cv2.imwrite('result.jpg',image)

You should be all set, except for the line for the face_cascade variable. We're going to grab a new xml file, which you can find here: Faces XML file

The way image recognition works is we first need to "train" a classifier, like we would with any machine learning algorithm. To do this, we generally need to compile a massive set of images of what we're looking to detect. In the case of faces, we'd want to grab 1,000 images of faces. Then we note where the faces are in the images. Then we feed this information to the machine, saying "hey, here are all the face images, there are the faces. Got it?" From here, you can either be all set, or sometimes you will take another step and show the machine a bunch of images that have no faces at all, and you tell the machine "there are no faces here! See no faces!" At this point, training is done. You're ready to show the machine a new image with a face. Hopefully, given it's "memory" of what images with faces were like, and what the actual faces in the images were like, our algorithm will be able to detect the face.

For a lot of the image recognition tasks, people have already built data sets for you to use for the training part. Face Detection is very popular, so there are already a lot of datasets for face data. License plates? Yep, popular. How about detecting dirty socks on the floor? It's unlikely there's a data set for that. This tutorial is not going to cover the creation of a data set. Keep your eyes peeled for a more in depth OpenCV tutorial series here that will.

Now, change the face_detection line in the script above to link to the Faces XML document I shared above. I put mine in the directory that I've been working in, just make sure you link to wherever you put yours:

face_cascade = cv2.CascadeClassifier('/home/pi/Desktop/GoPiGoLocal/faces.xml')

Run this script and smile at the camera! The script will either say it found some faces, or not. The resulting image will be output so you can see what it was seeing when it made the assessment. Here's an example from me:

Obviously, the next reasonable step from here would be to attempt to destroy all humans detected by firing missiles at them. Using our code from the foam USB missile launcher tutorial, let's go ahead and glue together OpenCV face detection and missile launches. What's the worst that could happen?

Here's a modified version:

'''
Much of this code comes from Dexter Industries' GoPiGo with office cannon script.

# http://www.dexterindustries.com/GoPiGo/                                                                
# History
# ------------------------------------------------
# Author     	Date      		Comments
# John Cole  	April 14  		Initial Authoring    
# Karan			27 June 14		Code cleanup and made more responsive   
# 				25 Aug  14		USB high current mode for Raspberry Pi Model B+ added        
#                                       


## License
 GoPiGo for the Raspberry Pi: an open source robotics platform for the Raspberry Pi.
 Copyright (C) 2015  Dexter Industries

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see .
'''


import io
import picamera
import cv2
import numpy
from gopigo import *

import struct
import os

import sys
import platform
import time
import socket
import re
import json
import urllib2
import base64

import usb.core
import usb.util

#Enable for Model B+ and disable for Model B
model_b_plus=True	# For the model B+ we need to turn this variable on to run the Office Cannon.
					# This can be left on for the Model B and not cause problems.
					# This pulls GPIO 38 to high, which overrides USB overcurrent setting.
					# With this set, the USB can deliver up to 1.2 Amps.
tdelay=80
# Protocol command bytes
DOWN    = 0x01
UP      = 0x02
LEFT    = 0x04
RIGHT   = 0x08
FIRE    = 0x10
STOP    = 0x20

DEVICE = None
DEVICE_TYPE = None

file = open( "/dev/input/mice", "rb" );
debug = 0

# Setup the Office Cannon
def setup_usb():
    global DEVICE 
    global DEVICE_TYPE

    DEVICE = usb.core.find(idVendor=0x2123, idProduct=0x1010)

    if DEVICE is None:
        DEVICE = usb.core.find(idVendor=0x0a81, idProduct=0x0701)
        if DEVICE is None:
            raise ValueError('Missile device not found')
        else:
            DEVICE_TYPE = "Original"
    else:
        DEVICE_TYPE = "Thunder"

    # On Linux we need to detach usb HID first
    if "Linux" == platform.system():
        try:
            DEVICE.detach_kernel_driver(0)
        except Exception, e:
            pass # already unregistered    
    DEVICE.set_configuration()

#Send command to the office cannon
def send_cmd(cmd):
    if "Thunder" == DEVICE_TYPE:
        DEVICE.ctrl_transfer(0x21, 0x09, 0, 0, [0x02, cmd, 0x00,0x00,0x00,0x00,0x00,0x00])
    elif "Original" == DEVICE_TYPE:
        DEVICE.ctrl_transfer(0x21, 0x09, 0x0200, 0, [cmd])

#Send command to control the LED on the office cannon
def led(cmd):
    if "Thunder" == DEVICE_TYPE:
        DEVICE.ctrl_transfer(0x21, 0x09, 0, 0, [0x03, cmd, 0x00,0x00,0x00,0x00,0x00,0x00])
    elif "Original" == DEVICE_TYPE:
        print("There is no LED on this device")

#Send command to move the office cannon
def send_move(cmd, duration_ms):
    send_cmd(cmd)
    time.sleep(duration_ms / 1000.0)
    send_cmd(STOP)

def run_command(command, value):
    command = command.lower()
    if command == "right":
        send_move(RIGHT, value)
    elif command == "left":
        send_move(LEFT, value)
    elif command == "up":
        send_move(UP, value)
    elif command == "down":
        send_move(DOWN, value)
    elif command == "zero" or command == "park" or command == "reset":
        # Move to bottom-left
        send_move(DOWN, 2000)
        send_move(LEFT, 8000)
    elif command == "pause" or command == "sleep":
        time.sleep(value / 1000.0)
    elif command == "led":
        if value == 0:
            led(0x00)
        else:
            led(0x01)
    elif command == "fire" or command == "shoot":
        if value < 1 or value > 4:
            value = 1
        # Stabilize prior to the shot, then allow for reload time after.
        time.sleep(0.5)
        for i in range(value):
            send_cmd(FIRE)
            time.sleep(4.5)
    else:
        print "Error: Unknown command: '%s'" % command

def run_command_set(commands):
    for cmd, value in commands:
        run_command(cmd, value)

#Read the values from the mouse
def getMouseEvent():
	buf = file.read(3)
	button = ord( buf[0] )
	bLeft = button & 0x1
	bMiddle = ( button & 0x4 ) > 0
	bRight = ( button & 0x2 ) > 0
	x,y = struct.unpack( "bb", buf[1:] )
	if debug:
		print ("L:%d, M: %d, R: %d, x: %d, y: %d\n" % (bLeft,bMiddle,bRight, x, y) )
	return [bLeft,bMiddle,bRight,x,y]
  
  
flag=0
#Control the office cannon
def control():
	global flag
	[bLeft,bMiddle,bRight,x,y]=getMouseEvent()  #Get the inputs from the mouse

	#GoPiGo control
	if flag==1: 			#If left or right mouse not pressed, move forward
		fwd()
		flag=0
	if bLeft:    			#If left mouse buton pressed, turn left
		left()
		flag=1
  
	if bRight:    			#If right mouse button presses, turn right
		right()
		flag=1
	if bLeft and bRight:  	#If both the left and right mouse buttons pressed, go back
		stop()
		flag=0
 
	#Office cannon control
	
	if bMiddle > 0:
		print "fire rockets"
		run_command("fire", tdelay)

	#Move the mouse left to move the cannon left
	#Move the mouse right to move the cannon right
	#Press middle button to fire
	
	if x == 0:
		print "Stop rockets"
	elif x > 10:
		print "Left rockets"
		run_command("left", tdelay)
	elif x < -10:
		print "Right rockets"
		run_command("right", tdelay)
	if y == 0:
		print "Stop Rockets Y"
	elif y > 10:
		print "Up Rockets"
		run_command("up", tdelay)
	elif y < -10:
		print "Down rockets"
		run_command("down", tdelay)

	time.sleep(.1)
	return


def key_input(event):
    key_press = event.keysym.lower()
    print(key_press)

    if key_press == 'w':
        fwd()
    elif key_press == 's':
        bwd()
    elif key_press == 'a':
        left()
    elif key_press == 'd':
        right()
    elif key_press == 'q':
        left_rot()
    elif key_press == 'e':
        right_rot()
    elif key_press == 'space':
        stop()
    elif key_press == 'u':
        print(us_dist(15))

    elif key_press.isdigit():
        if int(key_press) in servo_range:
            enable_servo()
            servo(int(key_press)*14)
            time.sleep(1)
            disable_servo()

    elif key_press == 'control_r' or key_press == 'control_l':
        print('Fire rockets!')
        run_command("fire", tdelay)

    elif key_press == 'left':
        print('Left rockets')
        run_command('left', tdelay)
    elif key_press == 'right':
        print('right rockets')
        run_command('right', tdelay)
    elif key_press == 'up':
        print('up rockets')
        run_command('up', tdelay)
    elif key_press == 'down':
        print('down rockets')
        run_command('down', tdelay)


setup_usb()

#Enable USB to give supply upto 1.2A on model B+
if model_b_plus:
        os.system("gpio -g write 38 0")
        os.system("gpio -g mode 38 out")
        os.system("gpio -g write 38 1")
stop()  

raw_input("press enter to continue")
#Create a memory stream so photos doesn't need to be saved in a file
stream = io.BytesIO()

#Get the picture (low resolution, so it should be quite fast)
#Here you can also specify other parameters (e.g.:rotate the image)
with picamera.PiCamera() as camera:
    camera.resolution = (320, 240)
    camera.capture(stream, format='jpeg')

#Convert the picture into a numpy array
buff = numpy.fromstring(stream.getvalue(), dtype=numpy.uint8)

#Now creates an OpenCV image
image = cv2.imdecode(buff, 1)

#Load a cascade file for detecting faces
face_cascade = cv2.CascadeClassifier('/home/pi/Desktop/GoPiGoLocal/faces.xml')

#Convert to grayscale
gray = cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)

#Look for faces in the image using the loaded cascade file
faces = face_cascade.detectMultiScale(gray, 1.1, 5)

print "Found "+str(len(faces))+" face(s)"

if len(faces) > 0:
    run_command("fire", tdelay)


#Draw a rectangle around every found face
for (x,y,w,h) in faces:
    cv2.rectangle(image,(x,y),(x+w,y+h),(255,255,0),2)

#Save the result image
cv2.imwrite('result.jpg',image)

Now, if there's a face detected, missiles will be launched.

That's all for now, for the next tutorial: Home Page