Welcome to part 9 of the Python Plays: Grand Theft Auto series, where our first goal is to create a self-driving car. In this tutorial, we're going to cover how we can build a training dataset for a deep learning neural network.

If you're not fully up to speed with machine learning, I have a machine learning tutorial series, where you can also just mainly focus on the deep learning with Neural Networks and TensorFlow parts.

For training data, we need both features and labels. The featuresets will be the pixel data from the screen data. The labels are the key inputs we want the AI to make. We already have the pixel data, but we do not have a way of collecting inputs. We made inputs, but we need a way to capture our own inputs.

I researched for a way to log Python keys without the Python window being in focus, but came up empty-handed for anything that worked. I then took to Twitter to ask if anyone knew of a way, and Jake B (Their Github) offered a solution. I tweaked it slightly to suit my needs, and the result is the following, saved as getkeys.py:

# getkeys.py
# Citation: Box Of Hats (https://github.com/Box-Of-Hats )

import win32api as wapi
import time

keyList = ["\b"]
for char in "ABCDEFGHIJKLMNOPQRSTUVWXYZ 123456789,.'APS$/\\":
    keyList.append(char)

def key_check():
    keys = []
    for key in keyList:
        if wapi.GetAsyncKeyState(ord(key)):
            keys.append(key)
    return keys

Now we can just import the key_check function and run that any time we want to poll what keys are being pressed. Perfect, we have all the tools we need to do the job of training data collection, now we just need to put them together and to get some training data!

This part involves a lot of trial and error. This isn't some totally "solved" problem, it's open ended. Everything we do here can probably be improved upon. I will go over at least what I discovered along the way. To begin, let's create a new file, called create_training_data.py:

import numpy as np
from grabscreen import grab_screen
import cv2
import time
from getkeys import key_check
import os


def keys_to_output(keys):
    '''
    Convert keys to a ...multi-hot... array

    [A,W,D] boolean values.
    '''
    output = [0,0,0]
    
    if 'A' in keys:
        output[0] = 1
    elif 'D' in keys:
        output[2] = 1
    else:
        output[1] = 1
    return output

Immediately in our first major function here, we're bringing in a major choice. First, we're going to use a one-hot array to identify our move choice. We can either just be going forward, or we can be turning left, or we can be turning right.

This immediately is ignoring combinations of moves like speeding up and turning, or slowing down and turning...etc.

I've elected to do it this way to try to collect as much data as possible in as short of a time as possible. It turns out that driving nice in Grand Theft Auto V is fairly boring. I do not plan for our Agent to always be so well behaved, but, for now, let's keep things under control.

file_name = 'training_data.npy'

if os.path.isfile(file_name):
    print('File exists, loading previous data!')
    training_data = list(np.load(file_name))
else:
    print('File does not exist, starting fresh!')
    training_data = []

File does not exist, starting fresh!

Above, we're starting our training dataset. I wasn't sure what exactly I wanted to do for training data, but I did the math, and elected to just stuff it all into a numpy file, it would wind up taking more training samples than I was ready to create to necessitate a database or something. Eventually, however, we really would need a database of some kind if we wanted to have a flawless AI, since it would need huge amounts of training data. Feel free to change whatever you like along the way, I do my best to compartmentalize everything, so we can easily swap out new methodologies, or so you can without me.

Now, the same as before, we want to have a counter more than ever, since we don't want to be saving frames that aren't actually intended to be training data!

def main():

    for i in list(range(4))[::-1]:
        print(i+1)
        time.sleep(1)
        
    while(True):
        # 800x600 windowed mode
        screen = grab_screen(region=(0,40,800,640))
        last_time = time.time()
        screen = cv2.cvtColor(screen, cv2.COLOR_BGR2GRAY)
        # resize to something a bit more acceptable for a CNN
        screen = cv2.resize(screen, (80,60))
        keys = key_check()
        output = keys_to_output(keys)
        training_data.append([screen,output])
        
        if cv2.waitKey(25) & 0xFF == ord('q'):
            cv2.destroyAllWindows()
            break

        if len(training_data) % 500 == 0:
            print(len(training_data))
            np.save(file_name,training_data)

Above, we've got the countdown, and then we're collecting data. We're just appending everything to one huge Python list, and then, every 500 data points, we're saving to the Numpy file.

The way things are right now, you can play as long as you like, then, once you see it print out a new set of 500 datapoints, you can tab out of the game, stop the script, and your data will remain clean. When you start again, you have the timer to setup, so you should be all set.

Finally, note that we're resizing the game data to 80x60. This is so that the game data more easily fits into our Convolutional Neural Network when it comes time to train. As usual, feel free to tweak these values, but 80x60 is enough to get started. Alright, with this, hop to it! Create some data! I recommend you shoot for ~100K inputs.