Name		Name	Last commit message	Last commit date
metrics		metrics
models		models
quantizers		quantizers
report		report
.gitignore		.gitignore
README.md		README.md
benchmark.py		benchmark.py
prompt.py		prompt.py
sol_benchmark.sh		sol_benchmark.sh
utils.py		utils.py

Repository files navigation

MidLLaMAI

Steps & Dependencies:

Conda environment Python3.11

conda install nvidia/label/cuda-12.1.0::cuda

conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia

Then install whatever pip dependencies you get as complaints when running the below scripts.

For chatting: python prompt.py

For benchmarking: python benchmark.py

About

A benchmarking suite for evaluating performance of compressed LLaMA models focused on finding an optimal balance between different pruning and quantization techniques with varying number of model parameters

0 stars

1 watching

Report repository

Releases

Packages

Contributors

Languages