crazywoola
GitHub
parent
010ec79484
commit
a5479c106d
|
|
@ -4,56 +4,22 @@ With [OpenLLM](https://github.com/bentoml/OpenLLM), you can run inference with a
|
|||
And Dify supports connecting to OpenLLM deployed large language model's inference capabilities locally.
|
||||
|
||||
## Deploy OpenLLM Model
|
||||
|
||||
### Before you start
|
||||
|
||||
When using Docker to deploy a private model locally, you might need to access the service via the container's IP address instead of `127.0.0.1`. This is because `127.0.0.1` or `localhost` by default points to your host system and not the internal network of the Docker container. To retrieve the IP address of your Docker container, you can follow these steps:
|
||||
|
||||
1. First, determine the name or ID of your Docker container. You can list all active containers using the following command:
|
||||
|
||||
```bash
|
||||
docker ps
|
||||
```
|
||||
|
||||
2. Then, use the command below to obtain detailed information about a specific container, including its IP address:
|
||||
|
||||
```bash
|
||||
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' container_name_or_ID
|
||||
```
|
||||
|
||||
Please note that you usually do not need to manually find the IP address of the Docker container to access the service, because Docker offers a port mapping feature. This allows you to map the container ports to local machine ports, enabling access via your local address. For example, if you used the `-p 80:80` parameter when running the container, you can access the service inside the container by visiting `http://localhost:80` or `http://127.0.0.1:80`.
|
||||
|
||||
If you do need to use the container's IP address directly, the steps above will assist you in obtaining this information.
|
||||
|
||||
### Starting OpenLLM
|
||||
|
||||
Each OpenLLM Server can deploy one model, and you can deploy it in the following way:
|
||||
|
||||
1. First, install OpenLLM through PyPI:
|
||||
```bash
|
||||
docker run --rm -it -p 3333:3000 ghcr.io/bentoml/openllm start facebook/opt-1.3b --backend pt
|
||||
```
|
||||
|
||||
```bash
|
||||
$ pip install openllm
|
||||
```
|
||||
> Note: Using the `facebook/opt-1.3b` model here is only for demonstration, and the effect may not be good. Please choose the appropriate model according to the actual situation. For more models, please refer to: [Supported Model List](https://github.com/bentoml/OpenLLM#-supported-models).
|
||||
|
||||
2. Locally deploy and start the OpenLLM model:
|
||||
|
||||
```bash
|
||||
$ openllm start opt --model_id facebook/opt-125m -p 3333
|
||||
2023-08-20T23:49:59+0800 [INFO] [cli] Prometheus metrics for HTTP BentoServer from "_service:svc" can be accessed at http://localhost:3333/metrics.
|
||||
2023-08-20T23:50:00+0800 [INFO] [cli] Starting production HTTP BentoServer from "_service:svc" listening on http://0.0.0.0:3333 (Press CTRL+C to quit)
|
||||
```
|
||||
|
||||
After OpenLLM starts, it provides API access service for the local port `3333`, the endpoint being: `http://127.0.0.1:3333`. Since the default 3000 port conflicts with Dify's WEB service, the port is changed to 3333 here.
|
||||
If you need to modify the host or port, you can view the help information for starting OpenLLM: `openllm start opt --model_id facebook/opt-125m -h`.
|
||||
|
||||
> Note: Using the `facebook/opt-125m` model here is only for demonstration, and the effect may not be good. Please choose the appropriate model according to the actual situation. For more models, please refer to: [Supported Model List](https://github.com/bentoml/OpenLLM#-supported-models).
|
||||
|
||||
3. After the model is deployed, use the connected model in Dify.
|
||||
After the model is deployed, use the connected model in Dify.
|
||||
|
||||
Fill in under `Settings > Model Providers > OpenLLM`:
|
||||
|
||||
- Model Name: `facebook/opt-125m`
|
||||
- Server URL: `http://127.0.0.1:3333`
|
||||
- Model Name: `facebook/opt-1.3b`
|
||||
- Server URL: `http://<Machine_IP>:3333` Replace with your machine IP address
|
||||
|
||||
Click "Save" and the model can be used in the application.
|
||||
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@
|
|||
|
||||
Clone the Dify source code to your local machine:
|
||||
|
||||
```Shell
|
||||
```bash
|
||||
git clone https://github.com/langgenius/dify.git
|
||||
```
|
||||
|
||||
|
|
@ -20,7 +20,7 @@ git clone https://github.com/langgenius/dify.git
|
|||
|
||||
Navigate to the docker directory in the Dify source code and execute the following command to start Dify:
|
||||
|
||||
```Shell
|
||||
```bash
|
||||
cd dify/docker
|
||||
docker compose up -d
|
||||
```
|
||||
|
|
@ -29,7 +29,7 @@ docker compose up -d
|
|||
|
||||
Deployment Results:
|
||||
|
||||
```Shell
|
||||
```bash
|
||||
[+] Running 7/7
|
||||
✔ Container docker-web-1 Started 1.0s
|
||||
✔ Container docker-redis-1 Started 1.1s
|
||||
|
|
@ -42,13 +42,13 @@ Deployment Results:
|
|||
|
||||
Finally, check if all containers are running successfully:
|
||||
|
||||
```Shell
|
||||
```bash
|
||||
docker compose ps
|
||||
```
|
||||
|
||||
This includes 3 business services: api / worker / web, and 4 underlying components: weaviate / db / redis / nginx.
|
||||
|
||||
```Shell
|
||||
```bash
|
||||
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
|
||||
docker-api-1 langgenius/dify-api:0.3.2 "/entrypoint.sh" api 4 seconds ago Up 2 seconds 80/tcp, 5001/tcp
|
||||
docker-db-1 postgres:15-alpine "docker-entrypoint.s…" db 4 seconds ago Up 2 seconds 0.0.0.0:5432->5432/tcp
|
||||
|
|
@ -58,3 +58,15 @@ docker-weaviate-1 semitechnologies/weaviate:1.18.4 "/bin/weaviate --hos…"
|
|||
docker-web-1 langgenius/dify-web:0.3.2 "/entrypoint.sh" web 4 seconds ago Up 3 seconds 80/tcp, 3000/tcp
|
||||
docker-worker-1 langgenius/dify-api:0.3.2 "/entrypoint.sh" worker 4 seconds ago Up 2 seconds 80/tcp, 5001/tcp
|
||||
```
|
||||
|
||||
### Upgrade Dify
|
||||
|
||||
Enter the docker directory of the dify source code and execute the following commands:
|
||||
|
||||
```bash
|
||||
cd dify/docker
|
||||
git pull origin main
|
||||
docker compose down
|
||||
docker compose pull
|
||||
docker compose up -d
|
||||
```
|
||||
|
|
@ -12,7 +12,7 @@
|
|||
|
||||
Clone Dify 源代码至本地
|
||||
|
||||
```Shell
|
||||
```bash
|
||||
git clone https://github.com/langgenius/dify.git
|
||||
```
|
||||
|
||||
|
|
@ -42,13 +42,13 @@ docker compose up -d
|
|||
|
||||
最后检查是否所有容器都正常运行：
|
||||
|
||||
```Shell
|
||||
```bash
|
||||
docker compose ps
|
||||
```
|
||||
|
||||
包括 3 个业务服务 `api / worker / web`，以及 4 个基础组件 `weaviate / db / redis / nginx`。
|
||||
|
||||
```Shell
|
||||
```bash
|
||||
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
|
||||
docker-api-1 langgenius/dify-api:0.3.2 "/entrypoint.sh" api 4 seconds ago Up 2 seconds 80/tcp, 5001/tcp
|
||||
docker-db-1 postgres:15-alpine "docker-entrypoint.s…" db 4 seconds ago Up 2 seconds 0.0.0.0:5432->5432/tcp
|
||||
|
|
@ -58,3 +58,15 @@ docker-weaviate-1 semitechnologies/weaviate:1.18.4 "/bin/weaviate --hos…"
|
|||
docker-web-1 langgenius/dify-web:0.3.2 "/entrypoint.sh" web 4 seconds ago Up 3 seconds 80/tcp, 3000/tcp
|
||||
docker-worker-1 langgenius/dify-api:0.3.2 "/entrypoint.sh" worker 4 seconds ago Up 2 seconds 80/tcp, 5001/tcp
|
||||
```
|
||||
|
||||
### Upgrade Dify
|
||||
|
||||
进入 dify 源代码的 docker 目录，按顺序执行以下命令：
|
||||
|
||||
```bash
|
||||
cd dify/docker
|
||||
git pull origin main
|
||||
docker compose down
|
||||
docker compose pull
|
||||
docker compose up -d
|
||||
```
|
||||
|
|
@ -7,21 +7,6 @@ Dify 支持以本地部署的方式接入 LocalAI 部署的大型语言模型推
|
|||
|
||||
### 使用前注意事项
|
||||
|
||||
当您使用 Docker 在本地部署一个私有模型时，您可能需要通过容器的 IP 地址而不是 `127.0.0.1` 来访问该服务。这是因为 `127.0.0.1` 或 `localhost` 默认指向您的主机系统，而不是 Docker 容器的内部网络。为了获取 Docker 容器的 IP 地址，您可以按照以下步骤操作：
|
||||
|
||||
1. 首先，确定您的 Docker 容器的名称或 ID。您可以使用以下命令列出所有运行中的容器：
|
||||
|
||||
```bash
|
||||
docker ps
|
||||
```
|
||||
|
||||
2. 然后，使用以下命令获取指定容器的详细信息，包括其 IP 地址：
|
||||
|
||||
```bash
|
||||
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' 容器名称或ID
|
||||
```
|
||||
|
||||
请注意，您通常不需要手动寻找 Docker 容器的 IP 地址来访问服务，因为 Docker 提供了端口映射功能，允许您将容器端口映射到本机端口，从而通过本机地址进行访问。例如，如果您在运行容器时使用了 `-p 80:80` 参数，那么您可以通过访问 `http://localhost:80` 或 `http://127.0.0.1:80` 来访问容器中的服务。
|
||||
|
||||
如果确实需要直接使用容器的 IP 地址，以上步骤将帮助您获取到这一信息。
|
||||
|
||||
|
|
|
|||
|
|
@ -4,57 +4,22 @@
|
|||
Dify 支持以本地部署的方式接入 OpenLLM 部署的大型语言模型的推理能力。
|
||||
|
||||
## 部署 OpenLLM 模型
|
||||
|
||||
### 使用前注意事项
|
||||
|
||||
当您使用 Docker 在本地部署一个私有模型时，您可能需要通过容器的 IP 地址而不是 `127.0.0.1` 来访问该服务。这是因为 `127.0.0.1` 或 `localhost` 默认指向您的主机系统，而不是 Docker 容器的内部网络。为了获取 Docker 容器的 IP 地址，您可以按照以下步骤操作：
|
||||
|
||||
1. 首先，确定您的 Docker 容器的名称或 ID。您可以使用以下命令列出所有运行中的容器：
|
||||
|
||||
```bash
|
||||
docker ps
|
||||
```
|
||||
|
||||
2. 然后，使用以下命令获取指定容器的详细信息，包括其 IP 地址：
|
||||
|
||||
```bash
|
||||
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' 容器名称或ID
|
||||
```
|
||||
|
||||
请注意，您通常不需要手动寻找 Docker 容器的 IP 地址来访问服务，因为 Docker 提供了端口映射功能，允许您将容器端口映射到本机端口，从而通过本机地址进行访问。例如，如果您在运行容器时使用了 `-p 80:80` 参数，那么您可以通过访问 `http://localhost:80` 或 `http://127.0.0.1:80` 来访问容器中的服务。
|
||||
|
||||
如果确实需要直接使用容器的 IP 地址，以上步骤将帮助您获取到这一信息。
|
||||
|
||||
### 开始部署
|
||||
|
||||
每个 OpenLLM Server 可以部署一个模型，您可以通过以下方式部署：
|
||||
您可以通过以下方式部署：
|
||||
|
||||
1. 首先通过 PyPI 安装 OpenLLM：
|
||||
```bash
|
||||
docker run --rm -it -p 3333:3000 ghcr.io/bentoml/openllm start facebook/opt-1.3b --backend pt
|
||||
```
|
||||
> 注意：此处使用 facebook/opt-1.3b 模型仅作为示例，效果可能不佳，请根据实际情况选择合适的模型，更多模型请参考：[支持的模型列表](https://github.com/bentoml/OpenLLM#-supported-models)。
|
||||
|
||||
```bash
|
||||
$ pip install openllm
|
||||
```
|
||||
|
||||
2. 本地部署并启动 OpenLLM 模型：
|
||||
|
||||
```bash
|
||||
$ openllm start opt --model_id facebook/opt-125m -p 3333
|
||||
2023-08-20T23:49:59+0800 [INFO] [cli] Prometheus metrics for HTTP BentoServer from "_service:svc" can be accessed at http://localhost:3333/metrics.
|
||||
2023-08-20T23:50:00+0800 [INFO] [cli] Starting production HTTP BentoServer from "_service:svc" listening on http://0.0.0.0:3333 (Press CTRL+C to quit)
|
||||
```
|
||||
|
||||
OpenLLM 启动后，为本机的 `3333` 端口提供 API 接入服务，端点为：`http://127.0.0.1:3333`，由于默认的 3000 端口与 Dify 的 WEB 服务冲突，这边修改为 3333 端口。
|
||||
如需修改 host 或 port，可查看 OpenLLM 启动的帮助信息：`openllm start opt --model_id facebook/opt-125m -h`。
|
||||
|
||||
> 注意：此处使用 facebook/opt-125m 模型仅作为示例，效果可能不佳，请根据实际情况选择合适的模型，更多模型请参考：[支持的模型列表](https://github.com/bentoml/OpenLLM#-supported-models)。
|
||||
|
||||
3. 模型部署完毕，在 Dify 中使用接入模型
|
||||
模型部署完毕，在 Dify 中使用接入模型
|
||||
|
||||
在 `设置 > 模型供应商 > OpenLLM` 中填入：
|
||||
|
||||
- 模型名称：`facebook/opt-125m`
|
||||
- 服务器 URL：`http://127.0.0.1:3333`
|
||||
|
||||
- 模型名称：`facebook/opt-1.3b`
|
||||
- 服务器 URL：`http://<Machine_IP>:3333` 替换成您的机器 IP 地址
|
||||
"保存" 后即可在应用中使用该模型。
|
||||
|
||||
本说明仅作为快速接入的示例，如需使用 OpenLLM 更多特性和信息，请参考：[OpenLLM](https://github.com/bentoml/OpenLLM)
|
||||
|
|
@ -6,23 +6,6 @@
|
|||
|
||||
### 使用前注意事项
|
||||
|
||||
当您使用 Docker 在本地部署一个私有模型时，您可能需要通过容器的 IP 地址而不是 `127.0.0.1` 来访问该服务。这是因为 `127.0.0.1` 或 `localhost` 默认指向您的主机系统，而不是 Docker 容器的内部网络。为了获取 Docker 容器的 IP 地址，您可以按照以下步骤操作：
|
||||
|
||||
1. 首先，确定您的 Docker 容器的名称或 ID。您可以使用以下命令列出所有运行中的容器：
|
||||
|
||||
```bash
|
||||
docker ps
|
||||
```
|
||||
|
||||
2. 然后，使用以下命令获取指定容器的详细信息，包括其 IP 地址：
|
||||
|
||||
```bash
|
||||
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' 容器名称或ID
|
||||
```
|
||||
|
||||
请注意，您通常不需要手动寻找 Docker 容器的 IP 地址来访问服务，因为 Docker 提供了端口映射功能，允许您将容器端口映射到本机端口，从而通过本机地址进行访问。例如，如果您在运行容器时使用了 `-p 80:80` 参数，那么您可以通过访问 `http://localhost:80` 或 `http://127.0.0.1:80` 来访问容器中的服务。
|
||||
|
||||
如果确实需要直接使用容器的 IP 地址，以上步骤将帮助您获取到这一信息。
|
||||
|
||||
### 开始部署
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue