diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..d21242a --- /dev/null +++ b/.gitignore @@ -0,0 +1,18 @@ +# Environments +.env +.venv +env/ +venv/ +ENV/ +env.bak/ +venv.bak/ + +# Pyre type checker +.pyre/ +.idea/' + +# Intellij IDEA Files +.idea/ +.ideaDataSources/ + +.vscode/ \ No newline at end of file diff --git a/en/.gitbook/assets/xinference-webpage.png b/en/.gitbook/assets/xinference-webpage.png new file mode 100644 index 0000000..6aeca25 Binary files /dev/null and b/en/.gitbook/assets/xinference-webpage.png differ diff --git a/en/SUMMARY.md b/en/SUMMARY.md index 1259622..9376a25 100644 --- a/en/SUMMARY.md +++ b/en/SUMMARY.md @@ -45,6 +45,7 @@ * [Model Configuration](advanced/model-configuration/README.md) * [Hugging Face](advanced/model-configuration/hugging-face.md) * [Replicate](advanced/model-configuration/replicate.md) + * [Xinference](advanced/model-configuration/xinference.md) * [More Integration](advanced/more-integration.md) ## use cases diff --git a/en/advanced/model-configuration/README.md b/en/advanced/model-configuration/README.md index 6b383aa..b1c0cf7 100644 --- a/en/advanced/model-configuration/README.md +++ b/en/advanced/model-configuration/README.md @@ -75,6 +75,7 @@ There are many third-party models on hosting type providers. Access models need * [Hugging Face](hugging-face.md). * [Replicate](replicate.md). +* [Xinference](xinference.md). ### Use model diff --git a/en/advanced/model-configuration/xinference.md b/en/advanced/model-configuration/xinference.md new file mode 100644 index 0000000..6c37d2d --- /dev/null +++ b/en/advanced/model-configuration/xinference.md @@ -0,0 +1,61 @@ +# Connecting to Xinference Local Deployed Models + +[Xorbits inference](https://github.com/xorbitsai/inference)is a powerful and versatile library designed to serve language, speech recognition, and multimodal models, and can even be used on laptops. It supports various models compatible with GGML, such as chatglm, baichuan, whisper, vicuna, orca, etc. +And Dify supports connecting to Xinference deployed large language model inference and embedding capabilities locally. + +## Deploy Xinference + +There are two ways to deploy Xinference, namely [local deployment](https://github.com/xorbitsai/inference/blob/main/README.md#local) and [distributed deployment](https://github.com/xorbitsai/inference/blob/main/README.md#distributed), here we take local deployment as an example. + +1. First, install Xinference via PyPI: + + ```bash + $ pip install "xinference[all]" + ``` + +2. Start Xinference locally: + + ```bash + $ xinference + 2023-08-20 19:21:05,265 xinference 10148 INFO Xinference successfully started. Endpoint: http://127.0.0.1:9997 + 2023-08-20 19:21:05,266 xinference.core.supervisor 10148 INFO Worker 127.0.0.1:37822 has been added successfully + 2023-08-20 19:21:05,267 xinference.deploy.worker 10148 INFO Xinference worker successfully started. + ``` + + Xinference will start a worker locally by default, with the endpoint: `http://127.0.0.1:9997`, and the default port is `9997`. + To modify the host or port, you can refer to xinference's help information: `xinference --help`. + +3. Create and deploy the model + + Visit `http://127.0.0.1:9997`, select the model and specification you need to deploy, and click `Create` to create and deploy the model, as shown below: + +
+ + As different models have different compatibility on different hardware platforms, please refer to [Xinference built-in models](https://inference.readthedocs.io/en/latest/models/builtin/index.html) to ensure the created model supports the current hardware platform. + +4. Obtain the model UID + + Return to the command line interface and enter: + + ```bash + $ xinference list + UID Type Name Format Size (in billions) Quantization + ------------------------------------ ------ ----------- -------- -------------------- -------------- + a9e4d530-3f4b-11ee-a9b9-e6608f0bd69a LLM vicuna-v1.3 ggmlv3 7 q2_K + ``` + + The first column is the model UID created in step 3, such as `a9e4d530-3f4b-11ee-a9b9-e6608f0bd69a` above. + +5. After the model is deployed, connect the deployed model in Dify. + + In `Settings > Model Providers > Xinference`, enter: + + - Model name: `vicuna-v1.3` + - Server URL: `http://127.0.0.1:9997` + - Model UID: `a9e4d530-3f4b-11ee-a9b9-e6608f0bd69a` + + Click "Save" to use the model in the dify application. + +Dify also supports using [Xinference builtin models](https://github.com/xorbitsai/inference/blob/main/README.md#builtin-models) as Embedding models, just select the Embeddings type in the configuration box. + +For more information about Xinference, please refer to: [Xorbits Inference](https://github.com/xorbitsai/inference) \ No newline at end of file diff --git a/zh_CN/.gitbook/assets/xinference-webpage.png b/zh_CN/.gitbook/assets/xinference-webpage.png new file mode 100644 index 0000000..f45e6e7 Binary files /dev/null and b/zh_CN/.gitbook/assets/xinference-webpage.png differ diff --git a/zh_CN/SUMMARY.md b/zh_CN/SUMMARY.md index 26c046a..b71796c 100644 --- a/zh_CN/SUMMARY.md +++ b/zh_CN/SUMMARY.md @@ -44,6 +44,7 @@ * [模型配置](advanced/model-configuration/README.md) * [接入 Hugging Face 上的开源模型](advanced/model-configuration/hugging-face.md) * [接入 Replicate 上的开源模型](advanced/model-configuration/replicate.md) + * [接入 Xinference 部署的本地模型](advanced/model-configuration/xinference.md) * [更多集成](advanced/more-integration.md) ## 使用案例 diff --git a/zh_CN/advanced/model-configuration/README.md b/zh_CN/advanced/model-configuration/README.md index 021ee98..43c7423 100644 --- a/zh_CN/advanced/model-configuration/README.md +++ b/zh_CN/advanced/model-configuration/README.md @@ -78,6 +78,7 @@ Dify 使用了 [PKCS1_OAEP](https://pycryptodome.readthedocs.io/en/latest/src/ci * [Hugging Face](hugging-face.md)。 * [Replicate](replicate.md)。 +* [Xinference](xinference.md)。 diff --git a/zh_CN/advanced/model-configuration/xinference.md b/zh_CN/advanced/model-configuration/xinference.md new file mode 100644 index 0000000..bd8b552 --- /dev/null +++ b/zh_CN/advanced/model-configuration/xinference.md @@ -0,0 +1,61 @@ +# 接入 Xinference 部署的本地模型 + +[Xorbits inference](https://github.com/xorbitsai/inference) 是一个强大且通用的分布式推理框架，旨在为大型语言模型、语音识别模型和多模态模型提供服务，甚至可以在笔记本电脑上使用。它支持多种与GGML兼容的模型,如 chatglm, baichuan, whisper, vicuna, orca 等。 +Dify 支持以本地部署的方式接入 Xinference 部署的大型语言模型推理和 embedding 能力。 + +## 部署 Xinference + +部署 Xinference 有两种方式，分别为[本地部署](https://github.com/xorbitsai/inference/blob/main/README_zh_CN.md#%E6%9C%AC%E5%9C%B0%E9%83%A8%E7%BD%B2)和[分布式部署](https://github.com/xorbitsai/inference/blob/main/README_zh_CN.md#%E5%88%86%E5%B8%83%E5%BC%8F%E9%83%A8%E7%BD%B2)，以下以本地部署为例。 + +1. 首先通过 PyPI 安装 Xinference： + + ```bash + $ pip install "xinference[all]" + ``` + +2. 本地部署方式启动 Xinference： + + ```bash + $ xinference + 2023-08-20 19:21:05,265 xinference 10148 INFO Xinference successfully started. Endpoint: http://127.0.0.1:9997 + 2023-08-20 19:21:05,266 xinference.core.supervisor 10148 INFO Worker 127.0.0.1:37822 has been added successfully + 2023-08-20 19:21:05,267 xinference.deploy.worker 10148 INFO Xinference worker successfully started. + ``` + + Xinference 默认会在本地启动一个 worker，端点为：`http://127.0.0.1:9997`，端口默认为 `9997`。 + 如需修改 host 或 port，可查看 xinference 的帮助信息：`xinference --help`。 + +3. 创建并部署模型 + + 进入 `http://127.0.0.1:9997` 选择需要部署的模型和规格，点击 `Create` 即可创建并部署模型，如下图所示： + +
+ + 由于不同模型在不同硬件平台兼容性不同，请查看 [Xinference 内置模型](https://inference.readthedocs.io/en/latest/models/builtin/index.html) 确定创建的模型是否支持当前硬件平台。 + +4. 获取模型 UID + + 回到命令行界面，输入： + + ```bash + $ xinference list + UID Type Name Format Size (in billions) Quantization + ------------------------------------ ------ ----------- -------- -------------------- -------------- + a9e4d530-3f4b-11ee-a9b9-e6608f0bd69a LLM vicuna-v1.3 ggmlv3 7 q2_K + ``` + + 第一列即为第 3 步创建的模型 UID，如上面为 `a9e4d530-3f4b-11ee-a9b9-e6608f0bd69a`。 + +5. 模型部署完毕，在 Dify 中使用接入模型 + + 在 `设置 > 模型供应商 > Xinference` 中填入： + + - 模型名称：`vicuna-v1.3` + - 服务器 URL：`http://127.0.0.1:9997` + - 模型 UID：`a9e4d530-3f4b-11ee-a9b9-e6608f0bd69a` + + "保存" 后即可在应用中使用该模型。 + +Dify 同时支持将 [Xinference embed 模型](https://github.com/xorbitsai/inference/blob/main/README_zh_CN.md#%E5%86%85%E7%BD%AE%E6%A8%A1%E5%9E%8B) 作为 Embedding 模型使用，只需在配置框中选择 `Embeddings` 类型即可。 + +如需获取 Xinference 更多信息，请参考：[Xorbits Inference](https://github.com/xorbitsai/inference/blob/main/README_zh_CN.md) \ No newline at end of file