Introduction to Machine Learning — Assignment Portfolio

Course: Introduction to Machine Learning
Institution: VNUHCM — University of Science (HCMUS)
Author: Nguyễn Thế Hiển

Repository: end-to-end homework implementations from classical linear models to multilayer neural networks

This repository collects four consecutive programming assignments that build a coherent path through core supervised learning: closed-form linear regression, multiclass Softmax regression with gradient methods, feature/geometry-aware preprocessing, and feed-forward neural networks with regularization and post-training quantization. Implementations follow the course lecture notes and assignment specifications, with an emphasis on mathematical fidelity, reproducible NumPy code, and clear experimental reporting.

---

Learning trajectory

Stage	Topic	Core idea
HW1	Linear Regression	Closed-form estimation; polynomial feature expansion
HW2	Softmax Regression	Multiclass classification via gradient descent on MNIST
HW3	Softmax + SGD + Deslant	Faster optimization; geometric normalization of digits
HW4	Neural Networks	Backpropagation, dropout, int8 weight quantization

The sequence mirrors a standard introductory ML pipeline used in research-oriented courses (e.g., CS229-style foundations): start from a linear predictor, move to probabilistic multiclass models, improve optimization and representation, then introduce multilayer nonlinear models and practical deployment constraints.

---

Repository structure

Machine-Learning-/
├── README.md
├── HW1/
│   └── HW1.ipynb                 # Linear Regression
├── HW2/
│   ├── HW2.ipynb                 # Softmax Regression (full-batch GD)
│   ├── HW2-Slide.pdf
│   └── mnist.pkl.gz
├── HW3 - Softmax Regression/
│   ├── HW3.ipynb                 # Softmax + mini-batch SGD + deslant
│   ├── HW3-Slide.pdf
│   └── mnist.pkl.gz
└── HW4 - Neural Network/
    ├── HW4.ipynb                 # MLP + dropout + quantization
    ├── HW4-Slide.pdf
    └── mnist.pkl.gz

Note (HW1 data): train.txt and new.txt are required beside HW1.ipynb at runtime (provided with the assignment handout; not always stored in git).

---

Environment

• Language: Python 3.10+ (course environment: ml-env recommended)
• Core libraries: numpy, matplotlib
• Interface: Jupyter Notebook / JupyterLab / VS Code notebooks

# Example setup
python -m venv ml-env
source ml-env/bin/activate          # Windows: ml-env\Scripts\activate
pip install numpy matplotlib jupyter

For HW2–HW4, keep mnist.pkl.gz in the same folder as the notebook before running.

---

How to run

1. Open the target notebook (HW*.ipynb).
2. Ensure required data files are in that homework directory.
3. Run Kernel → Restart & Run All (required submission protocol for automated grading).
4. For Moodle submission: pack only the notebook as
   MSSV/HW*.ipynb → zip (e.g. 1234567/HW1.ipynb).

Long runs: HW3 full-batch GD comparison and HW4 100-epoch MLP training may take several minutes on CPU.

---

Assignment summaries

HW1 — Linear Regression

Goal. Learn a linear predictor (y \approx \mathbf{w}^\top \mathbf{z}) from synthetic supervised pairs, then improve fit via feature design.

Methods.

• Bias-augmented design matrix (add_ones)
• Closed-form solution with Moore–Penrose pseudoinverse:
  (\mathbf{w} = (X^\top X)^{+} X^\top y)
• Mean Absolute Error (MAE) for evaluation
• Degree-2 polynomial expansion (add_squares) to capture curvature when a hyperplane underfits

Data. Course-generated train.txt / new.txt ((d=2) features; code written for general (d \ge 1)).

Outcome. Linear features yield relatively large MAE; quadratic features reduce train/new MAE substantially (to about (8 \times 10^{-3})), illustrating the classic bias–capacity trade-off via feature maps rather than changing the learner.

---

HW2 — Softmax Regression (MNIST)

Goal. Multiclass digit classification ((K=10)) with Softmax regression trained by full-batch gradient descent.

Methods.

• Numerically stable Softmax forward pass
• Cross-entropy objective; GD updates on weight matrix (W \in \mathbb{R}^{(d+1)\times K})
• Learning-rate selection via training-curve comparison
• Mean Binary Error (MBE, % misclassification) for human-readable evaluation
• Hand-crafted geometric feature: vertical span from the first ink row to the centroid row of pixels (> 0.5), min–max normalized using training statistics only

Data. MNIST split in mnist.pkl.gz — train 50k / val 10k / test 10k, (28\times28) grayscale flattened to 784-D.

Representative results (notebook).

• Softmax + bias: train MBE ≈ 9.2%, val ≈ 8.4%
• Softmax + extra geometric feature: test MBE ≈ 8.5%

Takeaway. A linear multiclass model is a strong, interpretable baseline; modest feature engineering helps, but representation limits remain.

---

HW3 — Softmax Regression, Part II (SGD + Deslant)

Goal. Improve HW2 along two axes: (1) optimization efficiency and (2) input normalization that removes writing slant.

Methods.

• Mini-batch SGD for Softmax (shuffle indices each epoch; early stop on train MBE)
• Empirical comparison: full-batch GD vs SGD under a shared MBE target (SGD reaches the target in far fewer wall-clock seconds)
• Deslant preprocessing (assignment geometry):
  • Ink pixels: intensity (> 0.5)
  • Centroid ((\bar{r},\bar{c}))
  • Best-fit line (\text{column} = a\cdot\text{row} + b), with (\tan\alpha = -a)
  • Horizontal shear: (O[r,c] = I[r,; c + (\bar{r}-r)\tan\alpha]) with linear interpolation and column-index clamping to ([0,27])

Representative results (notebook).

• Softmax + SGD (no deslant): train MBE ≈ 6.8%, val ≈ 7.0%
• Softmax + SGD + deslant: train ≈ 4.7%, val ≈ 4.8%, test ≈ 5.2%

Takeaway. Better optimizers reduce training time; geometry-aware normalization reduces nuisance variation and improves generalization for a still-linear classifier.

---

HW4 — Neural Networks

Goal. Move from linear Softmax to a multilayer perceptron (MLP) with nonlinear hidden units; study overfitting control and model compression.

Methods.

• Forward: sigmoid hidden layers (with bias units), Softmax output
• Training: mini-batch SGD + backpropagation (output (\delta^{(L)}=P-Y); hidden (\delta) via chain rule through sigmoid)
• Dropout (inverted dropout on hidden activations) as anti-overfitting
• Model selection on validation MBE
• Post-training uniform int8 quantization of weights (scale = max|W| + 10^{-8}), dequantize at inference, same network topology

Architecture (default experiment). Input 784 → hidden 50 (sigmoid) → 10-way Softmax; 100 epochs, batch 32, learning rate 0.3 (as in the notebook protocol).

Representative results (notebook).

• MLP baseline: train MBE ≈ 0%, val ≈ 2.7% (strong lift over Softmax ~5%)
• MLP + dropout ((p=0.15)): validation competitive / improved vs baseline in controlled runs
• Quantized int8 weights: ~8× smaller parameter storage; validation MBE stays close to the float model when dequantization + forward pass match the trained architecture

Takeaway. Nonlinear hidden layers raise capacity; regularization and validation-based selection manage overfitting; quantization shows the accuracy–footprint trade-off relevant to deployment.

---

Methodological conventions

Across notebooks we consistently apply practices expected in academic ML coursework:

1. Math–code alignment — updates and preprocessing follow the course slides (closed form, Softmax gradients, SGD sketch, backprop deltas, deslant formulas).
2. Numerical stability — Softmax logits centered by row-wise max; log terms guarded where needed.
3. Vectorization — NumPy matrix ops preferred over Python element loops (loops only where the handout explicitly allows, e.g., per-image deslant).
4. Train / validation / test discipline — hyperparameters and model choice use validation; test is for final reporting only.
5. Leakage control — normalization statistics (e.g., feature min/max) estimated on training data and reused on val/test.
6. Reproducibility — fixed RNG seeds in SGD/MLP training cells where required by the autograder.

---

Results at a glance (MNIST digit recognition)

Approximate validation / test error rates from the completed notebooks (lower is better):

Model	Val MBE	Test MBE
Softmax (HW2 baseline)	~8.4%	—
Softmax + geometric feature (HW2)	~8.4%	~8.5%
Softmax + SGD (HW3)	~7.0%	—
Softmax + SGD + deslant (HW3)	~4.8%	~5.2%
MLP (HW4)	~2.7%	(selected final model)

This progression illustrates a standard empirical narrative: strong linear baselines → better optimization & preprocessing → nonlinear models.

---

Academic integrity

These notebooks are coursework submissions. Collaboration on ideas is allowed by course policy; code and written answers must be your own, with sources cited when consulted. Do not share solution notebooks. Automated checks and Restart & Run All are part of grading.

---

References & further reading

Course materials (primary). Lecture slides bundled in each homework folder (HW2-Slide.pdf, HW3-Slide.pdf, HW4-Slide.pdf).

Background (optional).

• Bishop, C. M. Pattern Recognition and Machine Learning.
• Goodfellow, I., Bengio, Y., & Courville, A. Deep Learning.
• Srivastava et al. (2014). Dropout: A Simple Way to Prevent Neural Networks from Overfitting. JMLR.
• CS231n notes on regularization and training practices: https://cs231n.github.io/neural-networks-2/
• MNIST: LeCun, Cortes, Burges — http://yann.lecun.com/exdb/mnist/

---

Author

Nguyễn Thế Hiển — Introduction to Machine Learning, VNUHCM — University of Science (HCMUS).

For questions about this portfolio, open an issue on the repository or contact the author via the course channel.