Welcome to part seven of the Deep Learning with Neural Networks and TensorFlow tutorials. We've been working on attempting to apply our recently-learned basic deep neural network on a dataset of our own. In the previous tutorial, we created the create_sentiment_featuresets.py
file, which will take our string sample data and convert it to vectors. Now, we're going to use this and incorporate it into our previous model.
The code we used from that tutorial:
import tensorflow as tf from tensorflow.examples.tutorials.mnist import input_data mnist = input_data.read_data_sets("/tmp/data/", one_hot = True) n_nodes_hl1 = 500 n_nodes_hl2 = 500 n_nodes_hl3 = 500 n_classes = 10 batch_size = 100 x = tf.placeholder('float', [None, 784]) y = tf.placeholder('float') def neural_network_model(data): hidden_1_layer = {'weights':tf.Variable(tf.random_normal([784, n_nodes_hl1])), 'biases':tf.Variable(tf.random_normal([n_nodes_hl1]))} hidden_2_layer = {'weights':tf.Variable(tf.random_normal([n_nodes_hl1, n_nodes_hl2])), 'biases':tf.Variable(tf.random_normal([n_nodes_hl2]))} hidden_3_layer = {'weights':tf.Variable(tf.random_normal([n_nodes_hl2, n_nodes_hl3])), 'biases':tf.Variable(tf.random_normal([n_nodes_hl3]))} output_layer = {'weights':tf.Variable(tf.random_normal([n_nodes_hl3, n_classes])), 'biases':tf.Variable(tf.random_normal([n_classes])),} l1 = tf.add(tf.matmul(data,hidden_1_layer['weights']), hidden_1_layer['biases']) l1 = tf.nn.relu(l1) l2 = tf.add(tf.matmul(l1,hidden_2_layer['weights']), hidden_2_layer['biases']) l2 = tf.nn.relu(l2) l3 = tf.add(tf.matmul(l2,hidden_3_layer['weights']), hidden_3_layer['biases']) l3 = tf.nn.relu(l3) output = tf.matmul(l3,output_layer['weights']) + output_layer['biases'] return output def train_neural_network(x): prediction = neural_network_model(x) cost = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits(prediction,y) ) optimizer = tf.train.AdamOptimizer().minimize(cost) hm_epochs = 10 with tf.Session() as sess: sess.run(tf.initialize_all_variables()) for epoch in range(hm_epochs): epoch_loss = 0 for _ in range(int(mnist.train.num_examples/batch_size)): epoch_x, epoch_y = mnist.train.next_batch(batch_size) _, c = sess.run([optimizer, cost], feed_dict={x: epoch_x, y: epoch_y}) epoch_loss += c print('Epoch', epoch, 'completed out of',hm_epochs,'loss:',epoch_loss) correct = tf.equal(tf.argmax(prediction, 1), tf.argmax(y, 1)) accuracy = tf.reduce_mean(tf.cast(correct, 'float')) print('Accuracy:',accuracy.eval({x:mnist.test.images, y:mnist.test.labels})) train_neural_network(x)
The above code predicted hand-written digits, but we're going to just stuff our own data in there and see what happens. The only things we need to change are how we feed the data in, and that's it. Converting the data to the format we needed was the majority of the work. To begin, we need to import the create_sentiment_featuresets
file we created in the previous tutorial:
from create_sentiment_featuresets import create_feature_sets_and_labels
Next, we can comment out the importing of the mnist input data. Then, we replace the mnist variable with:
train_x,train_y,test_x,test_y = create_feature_sets_and_labels('/home/psyber/Desktop/tf_tuts/pos.txt','/home/psyber/Desktop/tf_tuts/neg.txt')
I will also change the node counts per layer to 1500, instead of 500.
Now, do we need to change anything in the neural_network_model
? Nope, that stays the same. How about in train_neural_network
? Yep, we're going to have to change the references to the mnist data, in the training and testing, and we also need to do our own batching code. If you recall in the tutorial where we covered the deep neural network, we made use of the mnist.train.next_batch
functionality that was just built in for us. We don't have that here. Luckily, this is not a complex task.
Thus, within the neural_network_model
function, let's modify the epoch for loop:
for epoch in range(hm_epochs): epoch_loss = 0 i=0 while i < len(train_x): start = i end = i+batch_size batch_x = np.array(train_x[start:end]) batch_y = np.array(train_y[start:end]) _, c = sess.run([optimizer, cost], feed_dict={x: batch_x, y: batch_y}) epoch_loss += c i+=batch_size print('Epoch', epoch+1, 'completed out of',hm_epochs,'loss:',epoch_loss)
Notice here that we're just using i
to iterate through the data, and our range is wherever we are with i
, to the i+batch_size
value. That's really all we need to do to do batching. Now, with these batches, we do everything else the same.
Now, we just need to slightly modify the correct and accuracy vars
correct = tf.equal(tf.argmax(prediction, 1), tf.argmax(y, 1)) accuracy = tf.reduce_mean(tf.cast(correct, 'float')) print('Accuracy:',accuracy.eval({x:test_x, y:test_y}))
Now the full code is:
from create_sentiment_featuresets import create_feature_sets_and_labels import tensorflow as tf #from tensorflow.examples.tutorials.mnist import input_data import pickle import numpy as np train_x,train_y,test_x,test_y = create_feature_sets_and_labels('/path/to/pos.txt','/path/to/neg.txt') n_nodes_hl1 = 1500 n_nodes_hl2 = 1500 n_nodes_hl3 = 1500 n_classes = 2 batch_size = 100 hm_epochs = 10 x = tf.placeholder('float') y = tf.placeholder('float') hidden_1_layer = {'f_fum':n_nodes_hl1, 'weight':tf.Variable(tf.random_normal([len(train_x[0]), n_nodes_hl1])), 'bias':tf.Variable(tf.random_normal([n_nodes_hl1]))} hidden_2_layer = {'f_fum':n_nodes_hl2, 'weight':tf.Variable(tf.random_normal([n_nodes_hl1, n_nodes_hl2])), 'bias':tf.Variable(tf.random_normal([n_nodes_hl2]))} hidden_3_layer = {'f_fum':n_nodes_hl3, 'weight':tf.Variable(tf.random_normal([n_nodes_hl2, n_nodes_hl3])), 'bias':tf.Variable(tf.random_normal([n_nodes_hl3]))} output_layer = {'f_fum':None, 'weight':tf.Variable(tf.random_normal([n_nodes_hl3, n_classes])), 'bias':tf.Variable(tf.random_normal([n_classes])),} # Nothing changes def neural_network_model(data): l1 = tf.add(tf.matmul(data,hidden_1_layer['weight']), hidden_1_layer['bias']) l1 = tf.nn.relu(l1) l2 = tf.add(tf.matmul(l1,hidden_2_layer['weight']), hidden_2_layer['bias']) l2 = tf.nn.relu(l2) l3 = tf.add(tf.matmul(l2,hidden_3_layer['weight']), hidden_3_layer['bias']) l3 = tf.nn.relu(l3) output = tf.matmul(l3,output_layer['weight']) + output_layer['bias'] return output def train_neural_network(x): prediction = neural_network_model(x) cost = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits(prediction,y) ) optimizer = tf.train.AdamOptimizer(learning_rate=0.001).minimize(cost) with tf.Session() as sess: sess.run(tf.initialize_all_variables()) for epoch in range(hm_epochs): epoch_loss = 0 i=0 while i < len(train_x): start = i end = i+batch_size batch_x = np.array(train_x[start:end]) batch_y = np.array(train_y[start:end]) _, c = sess.run([optimizer, cost], feed_dict={x: batch_x, y: batch_y}) epoch_loss += c i+=batch_size print('Epoch', epoch+1, 'completed out of',hm_epochs,'loss:',epoch_loss) correct = tf.equal(tf.argmax(prediction, 1), tf.argmax(y, 1)) accuracy = tf.reduce_mean(tf.cast(correct, 'float')) print('Accuracy:',accuracy.eval({x:test_x, y:test_y})) train_neural_network(x)
This gives us about 62% accuracy. Feel free to test with many more layers, more nodes...etc. You will likely find that the impact is minimal, and getting out of the 60s is futile.
If you recall from the beginning on the deep learning series, I said it was mostly dataset sizes that were important. In the next tutorial, we're going to use a much larger dataset to see if that makes any significant difference.