Installing Oobabooga on ArchLinux

Budi SyahiddinPRO @zeon256

20 May 2023

3 min read

AI CUDA LLM Oobabooga

# Overview

This blogpost is mainly for my personal documentation but figured it might help those that are having issues installing Oobabooga on Archlinux.

This guide will go through how to install Oobabooga manually without using the one-click installer as that causes problems on ArchLinux. This guide will also show how to install GPTQ-for-LLaMA so you can run 4 bits quantized models.

For this guide, we will be using commit 50c70e2 for Oobabooga and commit 81fe867 for GPTQ-for-LLaMa.

# Step 1: Installing correct version of CUDA

Ensure that we have the correct version of CUDA installed. To check that, we can run nvcc --version. This is the output you might see.

Code

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

If you are using 11.8 or 11.7, congrats! If you using the correect version! However if you are not, you will need to uninstall CUDA and install the appropriate version. To do that do this

Code

pacman -R cuda cudnn 
curl https://archive.org/download/archlinux_pkg_cuda/cuda-11.8.0-1-x86_64.pkg.tar.zst -O
sudo pacman -U cuda-11.8.0-1-x86_64.pkg.tar.zst

Then restart your computer. Do note that running nvidia-smi may provide a different version of CUDA. Check using nvcc --version!

# Step 2: Installing Conda

From now on this should be almost the same as the installation instructions on the official README with slight modifications. I will be copying most of the instructions verbatim just in case things changes.

Code

curl -sL "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh" > "Miniconda3.sh" 
bash Miniconda3.sh

# Step 2: Create a new conda environment

Code

conda create -n textgen python=3.10.9
conda activate textgen
conda install -c conda-forge cudatoolkit-dev

# Step 3: Install Pytorch (This is where things are different)

Ensure that you select the correct version of pytorch from this website. If you are on CUDA 11.7, choose CUDA 11.7 and CUDA 11.8 if you are using CUDA 11.8. The instruction below is for CUDA 11.8.

Code

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

# Step 4: Install the web UI

Code

git clone https://github.com/oobabooga/text-generation-webui
cd text-generation-webui
git checkout 50c70e2
pip install -r requirements.txt

# Step 5: Install GCC11

You will need GCC11 for this to work.

NOTE: DO NOT DOWNGRADE YOUR CURRENT GCC. JUST INSTALL ANOTHER ONE! THIS MIGHT CAUSE YOUR PC TO NOT BOOT!

This step might take awhile. Be patient 💀

Code

yay -S gcc11

# Step 6: Install GPTQ-for-LLaMA

We will need to temporarily setup CXX and CC for this to work.

Code

cd text-generation-webui
mkdir repositories
git clone https://github.com/qwopqwop200/GPTQ-for-LLaMa.git -b cuda
cd GPTQ-for-LLaMa
git checkout 81fe867
export CXX=/usr/bin/gcc-11
export CC=/usr/bin/gcc-11
python setup_cuda.py install

# Step 7: Download a model

This tutorial will showcase TheBloke/Wizard-Vicuna-7B-Uncensored-GPTQ

Code

cd ../../
python ./download-model.py TheBloke/Wizard-Vicuna-7B-Uncensored-GPTQ

# Step 8: Running the model

Code

python server.py --model TheBloke_vicuna-7B-1.1-GPTQ-4bit-128g --wbits 4 --groupsize 128 --api --verbose --listen

# Done!

Here's a screenshot demonstrating the webui!