Comparison of fine-tuned convolutional neural networks with different parameters for image classification of emotionn
Stefan Zeidler
CS711
Professor Rohit Kate
Assignment 4
December 12, 2024
My first trained model used an input layer followed by rescaling and two convolution layers. Each convolution layer had a pooling layer. This was then followed by a dense layer, a dropout layer, and the final output layer. The first convolution layer used 64 filters with a filter size of 4x4, and the second used 128 filters with a filter size of 3x3. The pooling layers both used 2x2 pooling and the dropout layer used a .2 dropout value The dense layer had 1024 hidden nodes. The convolutional and dense layers used the rectifier activation function while the output layer used softmax to classify.
My second trained model also had an input layer with rescaling, and two convolution layers. However, this model used 32 filters for both layers, each with a size of 3x3. There were no pooling or dropout layers. The dense layer had 1024 hidden nodes. The convolutional and dense layers used same activation as the first model.
| Model | Test Accuracy |
|---|---|
| Model 1 | 0.352 |
| Model 2 | 0.347 |
The first model showed higher test accuracy than the second with .352 vs .347. However, the accuracies were very close (and better than chance). I believe this is because, while the first model had pooling and dropout to prevent over-fitting, it did have a higher number of filters. In contrast, the second model had far fewer filters but did not have pooling or dropout to prevent overfitting.
Accuracy by Epoch for each Model
| Epoch | Model 1 | Model 2 |
|---|---|---|
| 1 | 0.218 | 0.245 |
| 2 | 0.288 | 0.401 |
| 3 | 0.379 | 0.597 |
| 4 | 0.49 | 0.798 |
| 5 | 0.601 | 0.906 |
| 6 | 0.716 | 0.97 |
| 7 | 0.847 | 0.985 |
| 8 | 0.872 | 0.986 |
| 9 | 0.945 | 0.968 |
| 10 | 0.971 | 0.966 |
| 11 | 0.953 | 0.992 |
| 12 | 0.982 | 0.993 |
| 13 | 0.989 | 0.971 |
| 14 | 0.983 | 0.952 |
| 15 | 0.976 | |
| 16 | 0.99 | |
| 17 | 0.991 | |
| 18 | 0.986 | |
| 19 | 0.992 | |
| 20 | 0.987 | |
| 21 | 0.991 | |
| 22 | 0.989 | |
| 23 | 0.991 | |
| 24 | 0.989 |
Regarding the model training, model 2 converged in fewer epochs than model 1. For early stopping, loss was monitored with a patience of 3 and minimum delta of .01, so that if three consecutive epochs had a less than 1% change in loss, training stopped.
For the pretrained model, I used the EfficientNetV2B3 model with a GlobalAveragePooling2D layer in order to reduce the dimensions. I then used a dense layer with 1024 nodes and a rectifier activation function.
Accuracy by Epoch for Fine-Tuned Mode.
| Epoch | Fine Tuned Model |
|---|---|
| 1 | 0.668 |
| 2 | 0.771 |
| 3 | 0.869 |
| 4 | 0.896 |
| 5 | 0.904 |
| 6 | 0.935 |
| 7 | 0.945 |
| 8 | 0.949 |
| 9 | 0.965 |
| 10 | 0.971 |
| 11 | 0.975 |
| 12 | 0.976 |
| 13 | 0.97 |
| 14 | 0.982 |
| 15 | 0.981 |
| 16 | 0.982 |
| 17 | 0.982 |
| Model | Test Accuracy |
|---|---|
| Model 1 | 0.352 |
| Fine Tuned | 0.53 |
The fine-tuned model converged in fewer epochs than training model 1 but took more than training model 2. The same early stopping criteria was used as in the models above.
The accuracy of the fine-tuned model was much higher than the best of the two trained models at .53 vs .352. I believe this is because it was already pre-trained on a much larger dataset of images and then adapted to the specific task of labeling emotions. In contrast, my trained models used a much smaller dataset and did not have any other training for image labeling at all. I think that the pre-trained model may have been able to leverage a lot of already learned features for this task. The fine-tuned model also had a much more complex and in-depth architecture than my trained models.
To my surprise, the trained model correctly predicted two labels while the pre-trained model only predicted one correctly. I do not think that it can be concluded that the trained model is more accurate on such a small sample, however. What I found interesting was that, in the last 3 images, both models predicted the same class incorrectly, which leads me to believe that something in the training data affected both models in similar ways.
Both models also seemed to have a bias away from predicting the happy class. For instance, the second and third images both are of happy people, and I would argue that the third image shows someone very clearly happy. However, the trained model was very certain that the third image was fear, and the fine-tuned model was very certain that image two was sad. Also, the fine-tuned model was absolutely certain (at 1.0) that the last image showed sadness.
I believe that I have the advantage of context since I know what is happening in each of the images, so I can add that in to appropriately label each image. My chosen images also appear more ambiguous, at least to me, than the training images. The training images appear to show stock photos of people who are unsubtly portraying a single emotion. However, emotions can often come together. For instance, my second image shows mixture of sadness and happiness. Negative emotions, such as fear and disgust, or anger and fear may occur together. And in some instance, the person in the image is trying to disguise their emotions, such as in image 4.