LLM Exploration: Offline Deployment of Ollama and one-api Services

Preface

Previously, I deployed DeepSeek using Ollama on a Linux server.

This time, I'm deploying on a server without public internet (or rather, one with significant restrictions), and I've encountered some pitfalls. Documenting them here.

ollama

ollama naturally cannot use the online installation script.

According to the ollama documentation, first download the installation package on your local computer based on the server's operating system and CPU architecture.

curl -L https://ollama.com/download/ollama-linux-amd64.tgz -o ollama-linux-amd64.tgz

Then use tools like scp to upload it to the server.

scp ollama-linux-amd64.tgz server-address:/temp

After connecting to the server, extract and install by following the ollama documentation (see the first reference).

sudo tar -C /usr -xzf ollama-linux-amd64.tgz

At this point, you can already run the ollama program.

ollama serve

Then add it as a service, which is also the officially recommended approach by ollama for easier management.

sudo useradd -r -s /bin/false -U -m -d /usr/share/ollama ollama
sudo usermod -a -G ollama $(whoami)

Create a new ollama.service file under /etc/systemd/system.

[Unit]
Description=Ollama Service
After=network-online.target

[Service]
ExecStart=/usr/bin/ollama serve
User=ollama
Group=ollama
Restart=always
RestartSec=3
Environment="PATH=$PATH"

[Install]
WantedBy=default.target

Then enable the service.

sudo systemctl daemon-reload
sudo systemctl enable ollama

At this point, the installation of ollama is complete.

Model Deployment

An offline server cannot use ollama pull to fetch models.

You need to first download the model on your local machine, where you can run the ollama pull command.

Then, locate the model files and upload them to the server.

The general idea is like this, and the specifics will be introduced next.

Finding Local Model Files

By default, if there is no special configuration, ollama's model files are all located in ~/.ollama/models/blobs.

First, execute the command to see the path of a specified model. For example, to find the deepseek-r1:32b model:

ollama show deepseek-r1:32b --modelfile

Output after executing the command (excerpt):

FROM C:\Users\deali\.ollama\models\blobs\sha256-96c415656d377afbff962f6cdb2394ab092ccbcbaab4b82525bc4ca800fe8a49
TEMPLATE """{{- if .System }}{{ .System }}{{ end }}
{{- range $i, $_ := .Messages }}
{{- $last := eq (len (slice $.Messages $i)) 1}}
{{- if eq .Role "user" }}<｜User｜>{{ .Content }}
{{- else if eq .Role "assistant" }}<｜Assistant｜>{{ .Content }}{{- if not $last }}<｜end▁of▁sentence｜>{{- end }}
{{- end }}
{{- if and $last (ne .Role "assistant") }}<｜Assistant｜>{{- end }}
{{- end }}"""
PARAMETER stop <｜begin▁of▁sentence｜>
PARAMETER stop <｜end▁of▁sentence｜>
PARAMETER stop <｜User｜>
PARAMETER stop <｜Assistant｜>

You can see this line:

FROM C:\Users\deali\.ollama\models\blobs\sha256-96c415656d377afbff962f6cdb2394ab092ccbcbaab4b82525bc4ca800fe8a49

This is the local path where ollama downloaded the model.

Upload this file to the server.

Exporting the Modelfile

This file format is similar to a Dockerfile.

Use the following command to export it:

ollama show deepseek-r1:32b --modelfile > Modelfile

Then, this file should also be uploaded to the server.

Importing a Model on the Server

After uploading the model files and Modelfile, place them in the same directory.

First, rename the file for easier subsequent import.

mv sha256-96c415656d377afbff962f6cdb2394ab092ccbcbaab4b82525bc4ca800fe8a49 deepseek-r1_32b.gguf

Next, edit the Modelfile. Change the FROM line to match the renamed model file name.

FROM ./deepseek-r1_32b.gguf

Then, run the following command to import.

ollama create deepseek-r1:32b -f Modelfile

If everything goes as expected, the import will be successful. You can run ollama list to verify whether the model has been imported.

one-api

One API is an open-source LLM (Large Language Model) API management and distribution system. It aims to provide unified access to various large models through a standard OpenAI API format, working out of the box. It supports multiple mainstream large models, including the OpenAI ChatGPT series, Anthropic Claude series, Google PaLM2/Gemini series, Mistral series, ByteDance Doubao (豆包) large model, Baidu Wenxin Yiyan (文心一言) series models, Alibaba Tongyi Qianwen (通义千问) series models, iFlytek Spark (讯飞星火) cognitive large model, Zhipu ChatGLM series models, Tencent Hunyuan (混元) large model, and more.

Docker Deployment

one-api is built using Go's Gin framework, making it easy to deploy. I typically use Docker for deployment, so I won't elaborate on that part here.

services:
  db:
    image: mysql:8.1.0
    container_name: mysql
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: mysql-password
    volumes:
      - ./data:/var/lib/mysql
  one-api:
    image: justsong/one-api
    container_name: one-api
    restart: always
    ports:
      - "3000:3000"
    depends:
      - db
    environment:
      - SQL_DSN=root:mysql-password@tcp(db:3306)/one_api
      - TZ=Asia/Shanghai
      - TIKTOKEN_CACHE_DIR=/TIKTOKEN_CACHE_DIR
    volumes:
      - ./data:/data
      - ./TIKTOKEN_CACHE_DIR:/TIKTOKEN_CACHE_DIR

networks:
  default:
    name: one-api

Solving tiktoken Issues

The problem encountered is that it depends on the tiktoken library, which requires internet access to download token encoders.

The solution involves checking the error logs, for example:

one-api  | [FATAL] 2025/02/17 - 10:47:21 | relay/adaptor/openai/token.go:26 [InitTokenEncoders] failed to get gpt-3.5-turbo token encoder: Get "https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken": dial tcp 57.150.97.129:443: i/o timeout, if you are using in offline environment, please set TIKTOKEN_CACHE_DIR to use exsited files

This requires downloading from https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken.

First, download the file locally and then upload it to the server.

But this alone is not sufficient.

tiktoken only recognizes URLs by their SHA-1 hash.

Generate the SHA-1:

TIKTOKEN_URL=https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken
echo -n $TIKTOKEN_URL | sha1sum | head -c 40

You can also combine it into a single command:

echo -n "https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken" | sha1sum | head -c 40

In this command, echo -n is used to output the specified URL string (the -n parameter's function is to prevent appending a newline character at the end of the output), sha1sum calculates its SHA-1 hash value, and head -c 40 extracts the first 40 characters, which are the first 40 bits of the hash value.

The execution result is:

9b5ad71b2ce5302211f9c61530b329a4922fc6a4

Then rename the cl100k_base.tiktoken file to the output 9b5ad71b2ce5302211f9c61530b329a4922fc6a4.

In the docker-compose.yaml mentioned earlier, we have already specified the TIKTOKEN_CACHE_DIR environment variable.

Then place this 9b5ad71b2ce5302211f9c61530b329a4922fc6a4 file in the TIKTOKEN_CACHE_DIR directory.

If you encounter similar errors later, repeat the above steps until there are no more errors.

The version I currently use has only downloaded two encoders.

Adding Ollama Channel in OneApi

This can be a bit tricky due to Docker networking issues.

There are several approaches. One is to run the OneApi container in host network mode.
Another is to use the host.docker.internal address.

In both cases, the prerequisite is that Ollama's host is set to 0.0.0.0. For configuration details, you can refer to my previous article: LLM Exploration: Local Deployment of the DeepSeek-R1 Model.

When adding the channel, select Ollama as the type.

In the custom model section, enter the model we deployed: deepseek-r1:32b.

Then, set the proxy to http://host.docker.internal:11434.

Note: In a Linux environment, host.docker.internal may not work. However, you can directly use the host machine's IP address. For example, if the host machine's IP address is 192.168.1.100, you can use http://192.168.1.100:11434 in OneApi to access the Ollama service.

References

• https://github.com/ollama/ollama/blob/main/docs/linux.md
• https://stackoverflow.com/questions/76106366/how-to-use-tiktoken-in-offline-mode-computer
• https://www.cnblogs.com/cjdty/p/18659438
• https://zhuanlan.zhihu.com/p/20485169539