Openchat Installation

Update 2

Ok, the actual problem I found and was able to reproduce:

following the original github instructions I ran into dependency issues:

conda create -y --name openchat python=3.11
conda activate openchat

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

pip3 install ochat

and after much testing, I was able to install without conflicts by running these commands:

conda create -y --name openchat-1 python=3.11.5
conda activate openchat-1
pip install xformers==0.0.22 # this installs torch 2.0.1
pip install ochat
pip install torchaudio==2.0.2
pip install torchvision==0.15.2

so it was really an issue with torch, torchaudio, and torchvision versions that led to the dep conflicts.

Update

actually, I tried to reproduce this problem but xformer==0.0.22 wasn't the issue. here's all the stuff I typed in the terminal 😢 I'll update again for the actual solution.

  205  conda create -y --name openchat python=3.11.5
  206  conda activate openchat
  207  pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
  208  pip3 install ochat
  212  pip uninstall torch torchaudio torchvision
  213  pip install torch==2.1.0 torchaudio==2.1.0 torchvision==0.16.0
  218  pip install --upgrade xformers
  219  pip uninstall torch torchaudio torchvision
  220  pip install --upgrade xformers
  221  pip install torch==2.0.1
  222  pip install torch==2.1.0
  223  pip uninstall xformers
  224  pip install xformers==0.0.22
  229  pip check
  230  python -m ochat.serving.openai_api_server --model openchat/openchat_3.5

Original

Today I followed the instructions to install openchat 3.5. I tried to install it via anaconda and I ran into some dependency issues, and a warning about python version installed was 3.11.6 but xformers was built for 3.11.5. Here's what I did to achieve a working installation:

first set the python version to exactly 3.11.5:

conda create -y --name openchat python=3.11.5

then as usual per the github instructions:

conda activate openchat
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
pip3 install ochat

and now I find these dep errors:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
xformers 0.0.22.post7 requires torch==2.1.0, but you have torch 2.0.1 which is incompatible.
vllm 0.2.1.post1 requires xformers==0.0.22, but you have xformers 0.0.22.post7 which is incompatible.

and if I installed torch-2.1.0:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
vllm 0.2.1.post1 requires torch==2.0.1, but you have torch 2.1.0 which is incompatible.
vllm 0.2.1.post1 requires xformers==0.0.22, but you have xformers 0.0.22.post7 which is incompatible.

so now all I had to do was set xformers==0.0.22 since 0.0.22.post7 doesn't count:

pip uninstall xformers
pip install xformers==0.0.22

and now it runs with no errors, huge success!

$ pip check
No broken requirements found.
$ python -m ochat.serving.openai_api_server --model openchat/openchat_3.5
FlashAttention not found. Install it if you need to train models.
FlashAttention not found. Install it if you need to train models.
2023-11-09 03:58:46,624 INFO worker.py:1673 -- Started a local Ray instance.
(pid=45563) FlashAttention not found. Install it if you need to train models.
(pid=45563) FlashAttention not found. Install it if you need to train models.
INFO 11-09 03:58:49 llm_engine.py:72] Initializing an LLM engine with config: model='openchat/openchat_3.5', tokenizer='openchat/openchat_3.5', tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, quantization=None, seed=0)
(AsyncTokenizer pid=45563) Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO 11-09 04:00:10 llm_engine.py:207] # GPU blocks: 2726, # CPU blocks: 2048
INFO:     Started server process [45364]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://localhost:18888 (Press CTRL+C to quit)

I've raised the issue to openchat on github, so maybe it'll be fixed soon hehe!