ChatGLM-6B local deployment

Download the project code

git clone https://github.com/THUDM/ChatGLM-6B.git

Create a virtual environment. It is recommended to use conda for management.

conda create -n chatglm python==3.8

Install dependencies

conda activate chatglm
pip install -r requirements.txt
conda install cudatoolkit=11.7 -c nvidia

PS: If cudatoolkit is not installed, you will encounter the error RuntimeError: Library cudart is not initialized.

However, some users in the issues mention that this can be resolved by using the CPU to output a quantized model and then directly invoking it, though this has not been tried yet.

Issues address: https://github.com/THUDM/ChatGLM-6B/issues/115

Hardware Requirements

Modify the startup parameters

model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).half().cuda()

Use the quantized model

# Modify as needed, currently only supports 4/8 bit quantization
model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).quantize(4).half().cuda()

Download the model and start the Gradio interface

cd ChatGLM-6B
python web_demo.py

Running effect