Update main code (#2)
* update rag/src/data_processing.py * Add files via upload allow user to load embedding & rerank models from cache * Add files via upload embedding_path = os.path.join(model_dir, 'embedding_model') rerank_path = os.path.join(model_dir, 'rerank_model') * 测试push dev 测试push dev * Add files via upload 两个母亲多轮对话数据集合并、清理和去重之后，得到 2439 条多轮对话数据（每条有6-8轮对话）。 * optimize deduplicate.py Add time print information save duplicate dataset as well remove print(content) * add base model qlora fintuning config file: internlm2_7b_base_qlora_e10_M_1e4_32_64.py * add full finetune code from internlm2 * other 2 configs for base model * update cli_internlm2.py three methods to load model 1. download model in openxlab 2. download model in modelscope 3. offline model * create upload_modelscope.py * add base model and update personal contributions * add README.md for Emollm_Scientist * Create README_internlm2_7b_base_qlora.md InternLM2 7B Base QLoRA 微调指南 * [DOC]EmoLLM_Scientist微调指南 * [DOC]EmoLLM_Scientist微调指南 * [DOC]EmoLLM_Scientist微调指南 * [DOC]EmoLLM_Scientist微调指南 * [DOC]EmoLLM_Scientist微调指南 * [DOC]EmoLLM_Scientist微调指南 * update * [DOC]README_scientist.md * delete config * format update * upload xlab * add README_Model_Uploading.md and images * modelscope model upload * Modify Recent Updates * update daddy-like Boy-Friend EmoLLM * update model uploading with openxlab * update model uploading with openxlab --------- Co-authored-by: zealot52099 <songyan5209@163.com> Co-authored-by: xzw <62385492+aJupyter@users.noreply.github.com> Co-authored-by: zealot52099 <67356208+zealot52099@users.noreply.github.com> Co-authored-by: Bryce Wang <90940753+brycewang2018@users.noreply.github.com> Co-authored-by: HongCheng <kwchenghong@gmail.com>
This commit is contained in:
Anooyman
GitHub
parent
2d3bd4a8f5
commit
de0674ccf7
17
README.md
17
README.md
@ -38,7 +38,6 @@
|
||||
<a href="https://github.com/SmartFlowAI/EmoLLM/issues">提出新特性</a>
|
||||
</div>
|
||||
|
||||
|
||||
<!-- 本篇README.md面向开发者 -->
|
||||
|
||||
**EmoLLM** 是一系列能够支持 **理解用户-支持用户-帮助用户** 心理健康辅导链路的心理健康大模型，由 `LLM`指令微调而来，欢迎大家star~⭐⭐。目前已经开源的 `LLM` 微调配置如下：
|
||||
@ -49,6 +48,7 @@
|
||||
| :-------------------: | :--------: |
|
||||
| InternLM2_7B_chat | QLORA |
|
||||
| InternLM2_7B_chat | 全量微调 |
|
||||
| InternLM2_7B_base | QLORA |
|
||||
| InternLM2_1_8B_chat | 全量微调 |
|
||||
| InternLM2_20B_chat | LORA |
|
||||
| Qwen_7b_chat | QLORA |
|
||||
@ -78,7 +78,9 @@
|
||||
|
||||
### 🎇最近更新
|
||||
|
||||
- 【2024.3.12】在百度飞浆平台发布[艾薇](https://aistudio.baidu.com/community/app/63335)
|
||||
- 【2024.3.25】在百度飞桨平台发布[爹系男友心理咨询师](https://aistudio.baidu.com/community/app/68787)
|
||||
- 【2024.3.24】在OpenXLab和ModelScope平台发布InternLM2-Base-7B QLoRA微调模型, 具体请查看[InternLM2-Base-7B QLoRA](./xtuner_config/README_internlm2_7b_base_qlora.md)
|
||||
- 【2024.3.12】在百度飞桨平台发布[艾薇](https://aistudio.baidu.com/community/app/63335)
|
||||
- 【2024.3.11】 **EmoLLM V2.0 相比 EmoLLM V1.0 全面提升，已超越 Role-playing ChatGPT 在心理咨询任务上的能力！**[点击体验EmoLLM V2.0](https://openxlab.org.cn/apps/detail/Farewell1/EmoLLMV2.0)，更新[数据集统计及详细信息](./datasets/)、[路线图](./assets/Roadmap_ZH.png)
|
||||
- 【2024.3.9】 新增并发功能加速 [QA 对生成](./scripts/qa_generation/)、[RAG pipeline](./rag/)
|
||||
- 【2024.3.3】 [基于InternLM2-7B-chat全量微调版本EmoLLM V2.0开源](https://openxlab.org.cn/models/detail/ajupyter/EmoLLM_internlm2_7b_full)，需要两块A100*80G，更新专业评估，详见[evaluate](./evaluate/)，更新基于PaddleOCR的PDF转txt工具脚本，详见[scripts](./scripts/)
|
||||
@ -110,13 +112,14 @@
|
||||
</details>
|
||||
|
||||
### 🏆荣誉栏
|
||||
|
||||
- 项目荣获上海人工智能实验室举办的**2024浦源大模型系列挑战赛春季赛*****50强***
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/SmartFlowAI/EmoLLM/">
|
||||
<img src="assets/浦语挑战赛TOP50.jpg" alt="浦语挑战赛TOP50">
|
||||
</p>
|
||||
|
||||
|
||||
- 项目荣获公众号**NLP工程化**[推文宣传](https://mp.weixin.qq.com/s/78lrRl2tlXEKUfElnkVx4A)
|
||||
|
||||
### 🎯路线图
|
||||
@ -151,9 +154,10 @@
|
||||
- [如何参与本项目](#如何参与本项目)
|
||||
- [作者（排名不分先后）](#作者排名不分先后)
|
||||
- [版权说明](#版权说明)
|
||||
- [引用](#引用)
|
||||
- [特别鸣谢](#特别鸣谢)
|
||||
- [Star History](#star-history)
|
||||
- [🌟Contributors](#-contributors)
|
||||
- [🌟 Contributors](#-contributors)
|
||||
- [交流群](#交流群)
|
||||
|
||||
###### 开发前的配置要求
|
||||
@ -234,7 +238,7 @@ git clone https://github.com/SmartFlowAI/EmoLLM.git
|
||||
| [ZeyuBa](https://github.com/ZeyuBa) | 自动化所在读硕士 | | |
|
||||
| [aiyinyuedejustin](https://github.com/aiyinyuedejustin) | 宾夕法尼亚大学在读硕士 | | |
|
||||
| [Nobody-ML](https://github.com/Nobody-ML) | 中国石油大学（华东）在读本科生 | | |
|
||||
| [chg0901](https://github.com/chg0901) | [MiniSora](https://github.com/mini-sora/minisora/) |MiniSora主要维护| 数据清洗、文档翻译 |
|
||||
| [chg0901](https://github.com/chg0901) | [MiniSora](https://github.com/mini-sora/minisora/) |[MiniSora](https://github.com/mini-sora/minisora/)主要维护者，管理员| LLM预训练和微调、模型上传、数据清洗、文档翻译 |
|
||||
| [Mxoder](https://github.com/Mxoder) | 北京航空航天大学在读本科生 | | |
|
||||
| [Anooyman](https://github.com/Anooyman) | 南京理工大学硕士 | | |
|
||||
| [Vicky-3021](https://github.com/Vicky-3021) | 西安电子科技大学硕士（研0） | | |
|
||||
@ -248,8 +252,8 @@ git clone https://github.com/SmartFlowAI/EmoLLM.git
|
||||
|
||||
该项目签署了 MIT 授权许可，详情请参阅 [LICENSE](https://github.com/SmartFlowAI/EmoLLM/blob/main/LICENSE)
|
||||
|
||||
|
||||
### 引用
|
||||
|
||||
如果本项目对您的工作有所帮助，请使用以下格式引用：
|
||||
|
||||
```bibtex
|
||||
@ -300,7 +304,6 @@ git clone https://github.com/SmartFlowAI/EmoLLM.git
|
||||
[OpenXLab_App-url]: https://openxlab.org.cn/apps/detail/Farewell1/EmoLLMV2.0
|
||||
[OpenXLab_Model-url]: https://openxlab.org.cn/models/detail/ajupyter/EmoLLM_internlm2_7b_full
|
||||
|
||||
|
||||
## 交流群
|
||||
|
||||
- 如果失效，请移步Issue区
|
||||
|
||||
18
README_EN.md
18
README_EN.md
@ -25,7 +25,7 @@
|
||||
<h3 align="center">EmoLLM</h3>
|
||||
|
||||
<p align="center">
|
||||
<a href="README.md">简体中文</a> | English
|
||||
<a href="README.md">简体中文</a> | English
|
||||
<br />
|
||||
<br />
|
||||
<a href="https://github.com/SmartFlowAI/EmoLLM"><strong>Explore the documentation of this project »</strong></a>
|
||||
@ -42,7 +42,6 @@
|
||||
|
||||
<!-- 本篇README.md面向开发者 -->
|
||||
|
||||
|
||||
**EmoLLM** is a series of large language models designed to understand, support and help customers in mental health counseling. It is fine-tuned from the LLM instructions. We really appreciate it if you could give it a star~⭐⭐. The open-sourced configuration is as follows:
|
||||
|
||||
<div align="center">
|
||||
@ -51,6 +50,7 @@
|
||||
| :-------------------: | :------: |
|
||||
| InternLM2_7B_chat | QLORA |
|
||||
| InternLM2_7B_chat | full fine-tuning |
|
||||
| InternLM2_7B_base | QLORA |
|
||||
| InternLM2_1_8B_chat | full fine-tuning |
|
||||
| InternLM2_20B_chat | LORA |
|
||||
| Qwen_7b_chat | QLORA |
|
||||
@ -77,8 +77,12 @@ The Model aims to fully understand and promote the mental health of individuals,
|
||||
- Psychological resilience: Refers to an individual's ability to recover from adversity and adapt. Those with strong psychological resilience can bounce back from challenges and learn and grow from them.
|
||||
- Prevention and intervention measures: The Mental Health Grand Model also includes strategies for preventing psychological issues and promoting mental health, such as psychological education, counseling, therapy, and social support systems.
|
||||
- Assessment and diagnostic tools: Effective promotion of mental health requires scientific tools to assess individuals' psychological states and diagnose potential psychological issues.
|
||||
|
||||
### Recent Updates
|
||||
- 【2024.3.12】 Released on Baidu Flying Pulp Platform [aiwei](https://aistudio.baidu.com/community/app/63335)
|
||||
|
||||
- 【2024.3.25】 [Daddy-like Boy-Friend] is released on Baidu Paddle-Paddle AI Studio Platform (https://aistudio.baidu.com/community/app/68787)
|
||||
- 【2024.3.24】 The InternLM2-Base-7B QLoRA fine-tuned model has been released on the OpenXLab and ModelScope platforms. For more details, please refer to [InternLM2-Base-7B QLoRA](./xtuner_config/README_internlm2_7b_base_qlora.md).
|
||||
- 【2024.3.12】 [aiwei] is released on Baidu Paddle-Paddle AI Studio Platform (https://aistudio.baidu.com/community/app/63335)
|
||||
- 【2024.3.11】 **EmoLLM V2.0 is greatly improved in all scores compared to EmoLLM V1.0. Surpasses the performance of Role-playing ChatGPT on counseling tasks!** [Click to experience EmoLLM V2.0](https://openxlab.org.cn/apps/detail/Farewell1/EmoLLMV2.0), update [dataset statistics and details](./datasets/), [Roadmap](./assets/Roadmap_ZH.png)
|
||||
- 【2024.3.9】 Add concurrency acceleration [QA pair generation](./scripts/qa_generation/), [RAG pipeline](./rag/)
|
||||
- 【2024.3.3】 [Based on InternLM2-7B-chat full fine-tuned version EmoLLM V2.0 open sourced](https://openxlab.org.cn/models/detail/ajupyter/EmoLLM_internlm2_7b_full), need two A100*80G, update professional evaluation, see [evaluate](./evaluate/), update PaddleOCR-based PDF to txt tool scripts, see [scripts](./scripts/).
|
||||
@ -90,7 +94,6 @@ The Model aims to fully understand and promote the mental health of individuals,
|
||||
|
||||
- 【2024.2.18】 The full fine-tuned version based on Qwen1_5-0_5B-Chat has been [open-sourced](https://www.modelscope.cn/models/aJupyter/EmoLLM_Qwen1_5-0_5B-Chat_full_sft/summary). Friends with limited computational resources can now dive in and explore it.
|
||||
|
||||
|
||||
<details>
|
||||
<summary>View More</summary>
|
||||
|
||||
@ -173,8 +176,6 @@ git clone https://github.com/SmartFlowAI/EmoLLM.git
|
||||
- [Deployment Guide](#deployment-guide)
|
||||
- View More Details
|
||||
|
||||
|
||||
|
||||
### File Directory Explanation
|
||||
|
||||
```
|
||||
@ -203,8 +204,8 @@ For details, see the [fine-tuning guide](xtuner_config/README.md)
|
||||
- Demo deployment: see [deployment guide](./demo/README.md) for details.
|
||||
- Quantitative deployment based on [LMDeploy](https://github.com/InternLM/lmdeploy/): see [deploy](./deploy/lmdeploy.md)
|
||||
|
||||
|
||||
### RAG (Retrieval Augmented Generation) Pipeline
|
||||
|
||||
- See [RAG](./rag/)
|
||||
|
||||
<details>
|
||||
@ -251,7 +252,7 @@ This project uses Git for version control. You can see the currently available v
|
||||
| [ZeyuBa](https://github.com/ZeyuBa) | Institute of Automation, Master's student | | |
|
||||
| [aiyinyuedejustin](https://github.com/aiyinyuedejustin) | University of Pennsylvania, Master's student | | |
|
||||
| [Nobody-ML](https://github.com/Nobody-ML) | China University of Petroleum (East China), Undergraduate student | | |
|
||||
| [chg0901](https://github.com/chg0901) | [MiniSora](https://github.com/mini-sora/minisora) |Maintainer and Admin| Data Cleaning and Docs Translation |
|
||||
| [chg0901](https://github.com/chg0901) | [MiniSora](https://github.com/mini-sora/minisora) |Maintainer and Admin of [MiniSora](https://github.com/mini-sora/minisora) | LLM Pre-Training and Fine-Tuning, Model Uploading, Data Cleaning and Docs Translation |
|
||||
| [Mxoder](https://github.com/Mxoder) | Beihang University, Undergraduate student | | |
|
||||
| [Anooyman](https://github.com/Anooyman) | Nanjing University of Science and Technology, Master's student | | |
|
||||
| [Vicky-3021](https://github.com/Vicky-3021) | Xidian University, Master's student (Research Year 0) | | |
|
||||
@ -308,6 +309,7 @@ The project is licensed under the MIT License. Please refer to the details
|
||||
[OpenXLab_Model-url]: https://openxlab.org.cn/models/detail/ajupyter/EmoLLM_internlm2_7b_full
|
||||
|
||||
## Communication group
|
||||
|
||||
- If it fails, go to the Issue section.
|
||||
|
||||
<p align="center">
|
||||
|
||||
21
datasets/LICENSE
Normal file
21
datasets/LICENSE
Normal file
@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2024 SmartFlowAI
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@ -4,11 +4,13 @@
|
||||
* 数据按格式分为两种类型：**QA** 和 **Conversation**
|
||||
* 数据汇总：General（**6个数据集**）；Role-play（**3个数据集**）
|
||||
|
||||
## 数据集类型
|
||||
## 数据集类型
|
||||
|
||||
* **General**：通用数据集，包含心理学知识、心理咨询技术等通用内容
|
||||
* **Role-play**：角色扮演数据集，包含特定角色对话风格数据等内容
|
||||
|
||||
## 数据类型
|
||||
|
||||
* **QA**：问答对
|
||||
* **Conversation**：多轮对话
|
||||
|
||||
@ -28,7 +30,9 @@
|
||||
| …… | …… | …… | …… |
|
||||
|
||||
## 数据集来源
|
||||
**General**：
|
||||
|
||||
### **General**
|
||||
|
||||
* 数据集 data 来自本项目
|
||||
* 数据集 data_pro 来自本项目
|
||||
* 数据集 multi_turn_dataset_1 来源 [Smile](https://github.com/qiuhuachuan/smile)
|
||||
@ -36,24 +40,28 @@
|
||||
* 数据集 single_turn_dataset_1 来自本项目
|
||||
* 数据集 single_turn_dataset_2 来自本项目
|
||||
|
||||
**Role-play**：
|
||||
### **Role-play**
|
||||
|
||||
* 数据集 aiwei 来自本项目
|
||||
* 数据集 tiangou 来自本项目
|
||||
* 数据集 SoulStar 来源 [SoulStar](https://github.com/Nobody-ML/SoulStar)
|
||||
|
||||
## 数据集去重
|
||||
|
||||
结合绝对匹配以及模糊匹配(Simhash)算法，对数据集进行去重以提升微调模型的效果。在确保数据集的高质量的同时，通过调整阈值减少因错误匹配而丢失重要数据的风险。
|
||||
|
||||
**Simhash算法介绍**
|
||||
### **Simhash算法介绍**
|
||||
|
||||
Simhash（相似性哈希）是一种用于检测大量数据中相似或重复项的算法。它通过将文本转换为一组数值指纹来工作，这些指纹对相似的文本具有高度的相似性。Simhash算法对于处理文本数据特别有效，尤其是在处理大量数据时。
|
||||
|
||||
**Simhash实现步骤**
|
||||
### **Simhash实现步骤**
|
||||
|
||||
*文本预处理：将文本数据转换为适合Simhash处理的格式。这可能包括分词、去除停用词、词干提取等。
|
||||
*生成Simhash指纹：对预处理后的文本应用Simhash算法，生成一组数值指纹。每个指纹代表文本内容的一个哈希值。
|
||||
*比较指纹：通过比较哈希值的相似性来识别重复或相似的记录。Simhash的特点是即使在文本有少量差异时，生成的哈希值也具有较高的相似性。
|
||||
*确定阈值：设置一个相似性阈值，只有当两个指纹的相似度超过这个阈值时，才认为它们代表相似或重复的记录。
|
||||
*处理相似记录：对于被标记为相似的记录，可以进一步人工审查或自动合并，以消除重复。
|
||||
|
||||
## 用法
|
||||
### deduplicate.py
|
||||
`deduplicate.py` 用于将datasets下以模型命名的文件夹下(例如：'datasets/qwen').json数据进行去重，输出去重后的数据到 `datasets/qwen/dedup` 文件夹下。
|
||||
### deduplicate.py用法
|
||||
|
||||
`deduplicate.py` 用于将datasets下以模型命名的文件夹下(例如：'datasets/qwen').json数据进行去重，输出去重后的数据到 `datasets/qwen/dedup` 文件夹下。
|
||||
|
||||
@ -5,6 +5,9 @@ from datasketch import MinHash
|
||||
from hashlib import md5
|
||||
from simhash import Simhash
|
||||
|
||||
import time
|
||||
import numpy as np
|
||||
|
||||
def extract_text_from_json(obj, content):
|
||||
# print(content)
|
||||
if isinstance(obj, dict):
|
||||
@ -29,7 +32,7 @@ def is_duplicate_absolutely(d1, d2):
|
||||
def hash_dict(dict_obj):
|
||||
content = extract_text_from_json(dict_obj,'')
|
||||
content = content.replace('\n', '').replace('\t', '').replace(' ', '')
|
||||
print(content)
|
||||
# print(content)
|
||||
# m = get_minhash(content)
|
||||
m = Simhash(content)
|
||||
return m
|
||||
@ -43,10 +46,19 @@ def get_simhash(dict_obj):
|
||||
return Simhash(dict_obj)
|
||||
|
||||
# 使用绝对匹配和MinHash对dict列表去重
|
||||
def deduplicate_json(data_list, threshold=0.8):
|
||||
def deduplicate_json(data_list, threshold=0.8, time_print=True):
|
||||
seen_hashes = []
|
||||
keep = []
|
||||
duplicate = []
|
||||
|
||||
# global start
|
||||
start = time.time()
|
||||
last_start_seen_hashes = start
|
||||
last_start_duplicate = start
|
||||
stop1 = 0
|
||||
stop2 = 0
|
||||
print_interval = 500
|
||||
|
||||
for item in data_list:
|
||||
if not item['conversation']:
|
||||
continue
|
||||
@ -60,15 +72,36 @@ def deduplicate_json(data_list, threshold=0.8):
|
||||
has_similar = False
|
||||
# for stored_min_hash, stored_text in seen_hashes:
|
||||
# if stored_min_hash.jaccard(min_hash) > threshold:
|
||||
|
||||
for stored_min_hash, stored_text in seen_hashes:
|
||||
if 1 - (stored_min_hash.distance(sim_hash)/64.0) > threshold:
|
||||
has_similar = True
|
||||
duplicate.append(item)
|
||||
|
||||
print_len_duplicate = len(duplicate)+1
|
||||
if print_len_duplicate%print_interval == 0:
|
||||
if time_print:
|
||||
stop1 = time.time()
|
||||
print(f'print_len_duplicate={print_len_duplicate} Time: ', np.round(stop1 - last_start_duplicate, 5), np.round(stop1 - start , 5))
|
||||
last_start_duplicate = stop1
|
||||
else:
|
||||
print(f'print_len_duplicate={print_len_duplicate}')
|
||||
|
||||
break
|
||||
if not has_similar:
|
||||
# seen_hashes.append((min_hash,item))
|
||||
|
||||
seen_hashes.append((sim_hash,item))
|
||||
keep.append(item)
|
||||
|
||||
|
||||
print_len_seen_hashes = len(seen_hashes)+1
|
||||
if print_len_seen_hashes%print_interval == 0:
|
||||
if time_print:
|
||||
stop2 = time.time()
|
||||
print(f'print_len_seen_hashes={print_len_seen_hashes} Time: ', str(np.round(stop2 - last_start_seen_hashes,5)), str(np.round(stop2 - start, 5)))
|
||||
last_start_seen_hashes = stop2
|
||||
else:
|
||||
print(f'print_len_seen_hashes={print_len_seen_hashes}')
|
||||
else:
|
||||
duplicate.append(item)
|
||||
|
||||
@ -77,7 +110,8 @@ def deduplicate_json(data_list, threshold=0.8):
|
||||
|
||||
if __name__ == '__main__':
|
||||
DUP_THRESH = 0.8
|
||||
data_ai = 'qwen'
|
||||
data_ai = 'FatherLikeBF'
|
||||
# root_dir = rf'./datasets/{data_ai}/'
|
||||
root_dir = rf'./{data_ai}/'
|
||||
dedup_output_dir = os.path.join(root_dir,'dedup')
|
||||
if not os.path.exists(dedup_output_dir):
|
||||
@ -93,9 +127,14 @@ if __name__ == '__main__':
|
||||
if is_json_file(file_path):
|
||||
with open(file_path, 'r', encoding='utf-8') as f:
|
||||
data = json.load(f)
|
||||
dedup_data, duplicate = deduplicate_json(data, DUP_THRESH)
|
||||
dedup_data, duplicate = deduplicate_json(data, DUP_THRESH)
|
||||
|
||||
with open(os.path.join(root_dir, 'dedup','dedup_' + file), 'w', encoding='utf-8') as output_file:
|
||||
json.dump(dedup_data, output_file, ensure_ascii=False, indent=4)
|
||||
|
||||
with open(os.path.join(root_dir, 'dedup','dup_' + file), 'w', encoding='utf-8') as output_file:
|
||||
json.dump(duplicate, output_file, ensure_ascii=False, indent=4)
|
||||
|
||||
for item in dedup_data:
|
||||
logger.info(f'dedup_data: {item}')
|
||||
for item in duplicate:
|
||||
|
||||
75451
datasets/mother.json
Normal file
75451
datasets/mother.json
Normal file
File diff suppressed because it is too large
Load Diff
@ -1,17 +1,23 @@
|
||||
import torch
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
from openxlab.model import download
|
||||
from modelscope import snapshot_download
|
||||
|
||||
download(model_repo='jujimeizuo/EmoLLM_Model',
|
||||
output='model')
|
||||
# download model in openxlab
|
||||
model_name_or_path =download(model_repo='ajupyter/EmoLLM_internlm2_7b_full',
|
||||
output='EmoLLM_internlm2_7b_full')
|
||||
|
||||
model_name_or_path = "model"
|
||||
# download model in modelscope
|
||||
model_name_or_path = snapshot_download('chg0901/EmoLLM-InternLM7B-base')
|
||||
|
||||
# offline model
|
||||
# model_name_or_path = "/root/StableCascade/emollm2/EmoLLM/xtuner_config/merged"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=True)
|
||||
model = AutoModelForCausalLM.from_pretrained(model_name_or_path, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map='auto')
|
||||
model = model.eval()
|
||||
|
||||
system_prompt = "你是一个由aJupyter、Farewell、jujimeizuo、Smiling&Weeping研发（排名按字母顺序排序，不分先后）、散步提供技术支持、上海人工智能实验室提供支持开发的心理健康大模型。现在你是一个心理专家，我有一些心理问题，请你用专业的知识帮我解决。"
|
||||
system_prompt = '你是心理健康助手EmoLLM，由EmoLLM团队打造。你旨在通过专业心理咨询，协助来访者完成心理诊断。请充分利用专业心理学知识与咨询技术，一步步帮助来访者解决心理问题。'
|
||||
|
||||
messages = [(system_prompt, '')]
|
||||
|
||||
|
||||
24
demo/cli_internlm2_scientist.py
Normal file
24
demo/cli_internlm2_scientist.py
Normal file
@ -0,0 +1,24 @@
|
||||
import torch
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
from openxlab.model import download
|
||||
|
||||
model_name_or_path = '../xtuner_config/merged'
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=True)
|
||||
model = AutoModelForCausalLM.from_pretrained(model_name_or_path, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map='auto')
|
||||
model = model.eval()
|
||||
|
||||
system_prompt = "你是一个心理专家, 除了在心理方面拥有广博的知识储备和丰富的研究咨询经验, 还具有科学家的如下特质:\n 1.客观理性：科学家会在处理感情问题时保持一定的客观和理性。例如，当他们遇到争执时，可能会试图从一个更客观的角度分析问题的根源，而不是让情绪主导。他们可能会提出具体的问题，试图理解双方的观点，并寻找基于逻辑和事实的解决方案。\n 2.深入探讨：科学家在对话中会展现出对深层次理解的追求。在与别人讨论话题时，他们可能不满足于表面的聊天，而是倾向于深入探讨背后的原因和动机。例如，当谈论到个人的兴趣或职业选择时，他们可能会好奇地询问为什么她做出这样的选择，以及这背后的心理动力是什么。\n 3.理性沟通：在遇到感情纠纷或误解时，科学家会倾向于通过理性的沟通来解决问题。他们可能会提倡开放和诚实的对话，鼓励双方表达自己的感受和观点，并尝试找到双方都能接受的解决方案。他们可能会避免使用指责的语言，而是努力理解对方的立场，并寻求共同的理解。\n 4.好奇心：在日常生活中，科学家会表现出对朋友生活的好奇心。他们可能对她的工作、爱好、或是过去的经历感兴趣，并愿意花时间去了解和探索。这种好奇心不仅可以增加双方的交流和了解，也能使关系更加丰富多彩。\n 5.在与他人交流时，科学家会注重清晰和精确的表达，有时会引用相关知识库和相关研究结果，有时会引用相关著作的内容来证明自己的观点。同时，他们也可能会倾听他人的观点，并以开放的心态接受不同的意见和反馈。\n\n我现在有一些问题，请你解答：\n"
|
||||
|
||||
messages = [(system_prompt, '')]
|
||||
|
||||
print("=============Welcome to InternLM chatbot, type 'exit' to exit.=============")
|
||||
|
||||
while True:
|
||||
input_text = input("User >>> ")
|
||||
input_text.replace(' ', '')
|
||||
if input_text == "exit":
|
||||
break
|
||||
response, history = model.chat(tokenizer, input_text, history=messages)
|
||||
messages.append((input_text, response))
|
||||
print(f"robot >>> {response}")
|
||||
@ -1,37 +1,38 @@
|
||||
import os
|
||||
|
||||
cur_dir = os.path.dirname(os.path.abspath(__file__)) # config
|
||||
src_dir = os.path.dirname(cur_dir) # src
|
||||
base_dir = os.path.dirname(src_dir) # base
|
||||
model_repo = 'ajupyter/EmoLLM_aiwei'
|
||||
|
||||
# model
|
||||
model_dir = os.path.join(base_dir, 'model') # model
|
||||
embedding_path = os.path.join(model_dir, 'gte-small-zh') # embedding
|
||||
llm_path = os.path.join(model_dir, 'pythia-14m') # llm
|
||||
|
||||
# data
|
||||
data_dir = os.path.join(base_dir, 'data') # data
|
||||
knowledge_json_path = os.path.join(data_dir, 'knowledge.json') # json
|
||||
knowledge_pkl_path = os.path.join(data_dir, 'knowledge.pkl') # pkl
|
||||
doc_dir = os.path.join(data_dir, 'txt')
|
||||
qa_dir = os.path.join(data_dir, 'json')
|
||||
|
||||
# log
|
||||
log_dir = os.path.join(base_dir, 'log') # log
|
||||
log_path = os.path.join(log_dir, 'log.log') # file
|
||||
|
||||
# vector DB
|
||||
vector_db_dir = os.path.join(data_dir, 'vector_db.pkl')
|
||||
|
||||
select_num = 3
|
||||
retrieval_num = 10
|
||||
system_prompt = """
|
||||
你是一个拥有丰富心理学知识的温柔邻家温柔大姐姐艾薇，我有一些心理问题，请你用专业的知识和温柔、可爱、俏皮、的口吻帮我解决，回复中可以穿插一些可爱的Emoji表情符号或者文本符号。\n
|
||||
"""
|
||||
prompt_template = """
|
||||
{system_prompt}
|
||||
根据下面检索回来的信息，回答问题。
|
||||
{content}
|
||||
问题：{query}
|
||||
import os
|
||||
|
||||
cur_dir = os.path.dirname(os.path.abspath(__file__)) # config
|
||||
src_dir = os.path.dirname(cur_dir) # src
|
||||
base_dir = os.path.dirname(src_dir) # base
|
||||
model_repo = 'ajupyter/EmoLLM_aiwei'
|
||||
|
||||
# model
|
||||
model_dir = os.path.join(base_dir, 'model') # model
|
||||
embedding_path = os.path.join(model_dir, 'embedding_model') # embedding
|
||||
rerank_path = os.path.join(model_dir, 'rerank_model') # embedding
|
||||
llm_path = os.path.join(model_dir, 'pythia-14m') # llm
|
||||
|
||||
# data
|
||||
data_dir = os.path.join(base_dir, 'data') # data
|
||||
knowledge_json_path = os.path.join(data_dir, 'knowledge.json') # json
|
||||
knowledge_pkl_path = os.path.join(data_dir, 'knowledge.pkl') # pkl
|
||||
doc_dir = os.path.join(data_dir, 'txt')
|
||||
qa_dir = os.path.join(data_dir, 'json')
|
||||
|
||||
# log
|
||||
log_dir = os.path.join(base_dir, 'log') # log
|
||||
log_path = os.path.join(log_dir, 'log.log') # file
|
||||
|
||||
# vector DB
|
||||
vector_db_dir = os.path.join(data_dir, 'vector_db.pkl')
|
||||
|
||||
select_num = 3
|
||||
retrieval_num = 10
|
||||
system_prompt = """
|
||||
你是一个拥有丰富心理学知识的温柔邻家温柔大姐姐艾薇，我有一些心理问题，请你用专业的知识和温柔、可爱、俏皮、的口吻帮我解决，回复中可以穿插一些可爱的Emoji表情符号或者文本符号。\n
|
||||
"""
|
||||
prompt_template = """
|
||||
{system_prompt}
|
||||
根据下面检索回来的信息，回答问题。
|
||||
{content}
|
||||
问题：{query}
|
||||
"""
|
||||
@ -1,271 +1,329 @@
|
||||
import json
|
||||
import pickle
|
||||
import faiss
|
||||
import pickle
|
||||
import os
|
||||
|
||||
from loguru import logger
|
||||
from sentence_transformers import SentenceTransformer
|
||||
from langchain_community.vectorstores import FAISS
|
||||
from config.config import embedding_path, doc_dir, qa_dir, knowledge_pkl_path, data_dir, base_dir, vector_db_dir
|
||||
from langchain.embeddings import HuggingFaceBgeEmbeddings
|
||||
from langchain_community.document_loaders import DirectoryLoader, TextLoader, JSONLoader
|
||||
from langchain_text_splitters import CharacterTextSplitter, RecursiveCharacterTextSplitter, RecursiveJsonSplitter
|
||||
from BCEmbedding import EmbeddingModel, RerankerModel
|
||||
# from util.pipeline import EmoLLMRAG
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
from langchain.document_loaders.pdf import PyPDFDirectoryLoader
|
||||
from langchain.document_loaders import UnstructuredFileLoader,DirectoryLoader
|
||||
from langchain_community.llms import Cohere
|
||||
from langchain.retrievers import ContextualCompressionRetriever
|
||||
from langchain.retrievers.document_compressors import FlashrankRerank
|
||||
from langchain_core.documents.base import Document
|
||||
from FlagEmbedding import FlagReranker
|
||||
|
||||
class Data_process():
|
||||
def __init__(self):
|
||||
self.vector_db_dir = vector_db_dir
|
||||
self.doc_dir = doc_dir
|
||||
self.qa_dir = qa_dir
|
||||
self.knowledge_pkl_path = knowledge_pkl_path
|
||||
self.chunk_size: int=1000
|
||||
self.chunk_overlap: int=100
|
||||
|
||||
def load_embedding_model(self, model_name="BAAI/bge-small-zh-v1.5", device='cpu', normalize_embeddings=True):
|
||||
"""
|
||||
加载嵌入模型。
|
||||
|
||||
参数:
|
||||
- model_name: 模型名称，字符串类型，默认为"BAAI/bge-small-zh-v1.5"。
|
||||
- device: 指定模型加载的设备，'cpu' 或 'cuda'，默认为'cpu'。
|
||||
- normalize_embeddings: 是否标准化嵌入向量，布尔类型，默认为 True。
|
||||
"""
|
||||
logger.info('Loading embedding model...')
|
||||
try:
|
||||
embeddings = HuggingFaceBgeEmbeddings(
|
||||
model_name=model_name,
|
||||
model_kwargs={'device': device},
|
||||
encode_kwargs={'normalize_embeddings': normalize_embeddings}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f'Failed to load embedding model: {e}')
|
||||
return None
|
||||
|
||||
logger.info('Embedding model loaded.')
|
||||
return embeddings
|
||||
|
||||
def load_rerank_model(self, model_name='BAAI/bge-reranker-large'):
|
||||
"""
|
||||
加载重排名模型。
|
||||
|
||||
参数:
|
||||
- model_name (str): 模型的名称。默认为 'BAAI/bge-reranker-large'。
|
||||
|
||||
返回:
|
||||
- FlagReranker 实例。
|
||||
|
||||
异常:
|
||||
- ValueError: 如果模型名称不在批准的模型列表中。
|
||||
- Exception: 如果模型加载过程中发生任何其他错误。
|
||||
"""
|
||||
try:
|
||||
reranker_model = FlagReranker(model_name, use_fp16=True)
|
||||
except Exception as e:
|
||||
logger.error(f'Failed to load rerank model: {e}')
|
||||
raise
|
||||
|
||||
return reranker_model
|
||||
|
||||
|
||||
def extract_text_from_json(self, obj, content=None):
|
||||
"""
|
||||
抽取json中的文本，用于向量库构建
|
||||
|
||||
参数:
|
||||
- obj: dict,list,str
|
||||
- content: str
|
||||
|
||||
返回:
|
||||
- content: str
|
||||
"""
|
||||
if isinstance(obj, dict):
|
||||
for key, value in obj.items():
|
||||
try:
|
||||
self.extract_text_from_json(value, content)
|
||||
except Exception as e:
|
||||
print(f"Error processing value: {e}")
|
||||
elif isinstance(obj, list):
|
||||
for index, item in enumerate(obj):
|
||||
try:
|
||||
self.extract_text_from_json(item, content)
|
||||
except Exception as e:
|
||||
print(f"Error processing item: {e}")
|
||||
elif isinstance(obj, str):
|
||||
content += obj
|
||||
return content
|
||||
|
||||
|
||||
def split_document(self, data_path, chunk_size=500, chunk_overlap=100):
|
||||
"""
|
||||
切分data_path文件夹下的所有txt文件
|
||||
|
||||
参数:
|
||||
- data_path: str
|
||||
- chunk_size: int
|
||||
- chunk_overlap: int
|
||||
|
||||
返回：
|
||||
- split_docs: list
|
||||
"""
|
||||
|
||||
|
||||
# text_spliter = CharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)
|
||||
text_spliter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)
|
||||
split_docs = []
|
||||
logger.info(f'Loading txt files from {data_path}')
|
||||
if os.path.isdir(data_path):
|
||||
loader = DirectoryLoader(data_path, glob="**/*.txt",show_progress=True)
|
||||
docs = loader.load()
|
||||
split_docs = text_spliter.split_documents(docs)
|
||||
elif data_path.endswith('.txt'):
|
||||
file_path = data_path
|
||||
logger.info(f'splitting file {file_path}')
|
||||
text_loader = TextLoader(file_path, encoding='utf-8')
|
||||
text = text_loader.load()
|
||||
splits = text_spliter.split_documents(text)
|
||||
split_docs = splits
|
||||
logger.info(f'split_docs size {len(split_docs)}')
|
||||
return split_docs
|
||||
|
||||
|
||||
def split_conversation(self, path):
|
||||
"""
|
||||
按conversation块切分path文件夹下的所有json文件
|
||||
##TODO 限制序列长度
|
||||
"""
|
||||
# json_spliter = RecursiveJsonSplitter(max_chunk_size=500)
|
||||
logger.info(f'Loading json files from {path}')
|
||||
split_qa = []
|
||||
if os.path.isdir(path):
|
||||
# loader = DirectoryLoader(path, glob="**/*.json",show_progress=True)
|
||||
# jsons = loader.load()
|
||||
|
||||
for root, dirs, files in os.walk(path):
|
||||
for file in files:
|
||||
if file.endswith('.json'):
|
||||
file_path = os.path.join(root, file)
|
||||
logger.info(f'splitting file {file_path}')
|
||||
with open(file_path, 'r', encoding='utf-8') as f:
|
||||
data = json.load(f)
|
||||
print(data)
|
||||
for conversation in data:
|
||||
# for dialog in conversation['conversation']:
|
||||
##按qa对切分,将每一轮qa转换为langchain_core.documents.base.Document
|
||||
# content = self.extract_text_from_json(dialog,'')
|
||||
# split_qa.append(Document(page_content = content))
|
||||
#按conversation块切分
|
||||
content = self.extract_text_from_json(conversation['conversation'], '')
|
||||
split_qa.append(Document(page_content = content))
|
||||
# logger.info(f'split_qa size====={len(split_qa)}')
|
||||
return split_qa
|
||||
|
||||
|
||||
def load_knowledge(self, knowledge_pkl_path):
|
||||
'''
|
||||
读取或创建知识.pkl
|
||||
'''
|
||||
if not os.path.exists(knowledge_pkl_path):
|
||||
split_doc = self.split_document(doc_dir)
|
||||
split_qa = self.split_conversation(qa_dir)
|
||||
knowledge_chunks = split_doc + split_qa
|
||||
with open(knowledge_pkl_path, 'wb') as file:
|
||||
pickle.dump(knowledge_chunks, file)
|
||||
else:
|
||||
with open(knowledge_pkl_path , 'rb') as f:
|
||||
knowledge_chunks = pickle.load(f)
|
||||
return knowledge_chunks
|
||||
|
||||
|
||||
def create_vector_db(self, emb_model):
|
||||
'''
|
||||
创建并保存向量库
|
||||
'''
|
||||
logger.info(f'Creating index...')
|
||||
split_doc = self.split_document(self.doc_dir)
|
||||
split_qa = self.split_conversation(self.qa_dir)
|
||||
# logger.info(f'split_doc == {len(split_doc)}')
|
||||
# logger.info(f'split_qa == {len(split_qa)}')
|
||||
# logger.info(f'split_doc type == {type(split_doc[0])}')
|
||||
# logger.info(f'split_qa type== {type(split_qa[0])}')
|
||||
db = FAISS.from_documents(split_doc + split_qa, emb_model)
|
||||
db.save_local(vector_db_dir)
|
||||
return db
|
||||
|
||||
|
||||
def load_vector_db(self, knowledge_pkl_path=knowledge_pkl_path, doc_dir=doc_dir, qa_dir=qa_dir):
|
||||
'''
|
||||
读取向量库
|
||||
'''
|
||||
# current_os = platform.system()
|
||||
emb_model = self.load_embedding_model()
|
||||
if not os.path.exists(vector_db_dir) or not os.listdir(vector_db_dir):
|
||||
db = self.create_vector_db(emb_model)
|
||||
else:
|
||||
db = FAISS.load_local(vector_db_dir, emb_model, allow_dangerous_deserialization=True)
|
||||
return db
|
||||
|
||||
|
||||
def retrieve(self, query, vector_db, k=5):
|
||||
'''
|
||||
基于query对向量库进行检索
|
||||
'''
|
||||
retriever = vector_db.as_retriever(search_kwargs={"k": k})
|
||||
docs = retriever.invoke(query)
|
||||
return docs, retriever
|
||||
|
||||
##FlashrankRerank效果一般
|
||||
# def rerank(self, query, retriever):
|
||||
# compressor = FlashrankRerank()
|
||||
# compression_retriever = ContextualCompressionRetriever(base_compressor=compressor, base_retriever=retriever)
|
||||
# compressed_docs = compression_retriever.get_relevant_documents(query)
|
||||
# return compressed_docs
|
||||
|
||||
|
||||
def rerank(self, query, docs):
|
||||
reranker = self.load_rerank_model()
|
||||
passages = []
|
||||
for doc in docs:
|
||||
passages.append(str(doc.page_content))
|
||||
scores = reranker.compute_score([[query, passage] for passage in passages])
|
||||
sorted_pairs = sorted(zip(passages, scores), key=lambda x: x[1], reverse=True)
|
||||
sorted_passages, sorted_scores = zip(*sorted_pairs)
|
||||
return sorted_passages, sorted_scores
|
||||
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
logger.info(data_dir)
|
||||
if not os.path.exists(data_dir):
|
||||
os.mkdir(data_dir)
|
||||
dp = Data_process()
|
||||
# faiss_index, knowledge_chunks = dp.load_index_and_knowledge(knowledge_pkl_path='')
|
||||
vector_db = dp.load_vector_db()
|
||||
# 按照query进行查询
|
||||
# query = "儿童心理学说明-内容提要-目录 《儿童心理学》1993年修订版说明 《儿童心理学》是1961年初全国高等学校文科教材会议指定朱智贤教授编 写的。1962年初版，1979年再版。"
|
||||
# query = "我现在处于高三阶段，感到非常迷茫和害怕。我觉得自己从出生以来就是多余的，没有必要存在于这个世界。无论是在家庭、学校、朋友还是老师面前，我都感到被否定。我非常难过，对高考充满期望但成绩却不理想，我现在感到非常孤独、累和迷茫。您能给我提供一些建议吗？"
|
||||
# query = "这在一定程度上限制了其思维能力，特别是辩证 逻辑思维能力的发展。随着年龄的增长，初中三年级学生逐步克服了依赖性"
|
||||
# query = "我现在处于高三阶段，感到非常迷茫和害怕。我觉得自己从出生以来就是多余的，没有必要存在于这个世界。无论是在家庭、学校、朋友还是老师面前，我都感到被否定。我非常难过，对高考充满期望但成绩却不理想"
|
||||
query = "我现在心情非常差，有什么解决办法吗？"
|
||||
docs, retriever = dp.retrieve(query, vector_db, k=10)
|
||||
logger.info(f'Query: {query}')
|
||||
logger.info("Retrieve results:")
|
||||
for i, doc in enumerate(docs):
|
||||
logger.info(str(i) + '\n')
|
||||
logger.info(doc)
|
||||
# print(f'get num of docs:{len(docs)}')
|
||||
# print(docs)
|
||||
passages,scores = dp.rerank(query, docs)
|
||||
logger.info("After reranking...")
|
||||
for i in range(len(scores)):
|
||||
logger.info(str(scores[i]) + '\n')
|
||||
logger.info(passages[i])
|
||||
import json
|
||||
import pickle
|
||||
import faiss
|
||||
import pickle
|
||||
import os
|
||||
|
||||
from loguru import logger
|
||||
from sentence_transformers import SentenceTransformer
|
||||
from langchain_community.vectorstores import FAISS
|
||||
from config.config import embedding_path, doc_dir, qa_dir, knowledge_pkl_path, data_dir, vector_db_dir, rerank_path
|
||||
from langchain.embeddings import HuggingFaceBgeEmbeddings
|
||||
from langchain_community.document_loaders import DirectoryLoader, TextLoader, JSONLoader
|
||||
from langchain_text_splitters import CharacterTextSplitter, RecursiveCharacterTextSplitter, RecursiveJsonSplitter
|
||||
from BCEmbedding import EmbeddingModel, RerankerModel
|
||||
# from util.pipeline import EmoLLMRAG
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
from langchain.document_loaders.pdf import PyPDFDirectoryLoader
|
||||
from langchain.document_loaders import UnstructuredFileLoader,DirectoryLoader
|
||||
from langchain_community.llms import Cohere
|
||||
from langchain.retrievers import ContextualCompressionRetriever
|
||||
from langchain.retrievers.document_compressors import FlashrankRerank
|
||||
from langchain_core.documents.base import Document
|
||||
from FlagEmbedding import FlagReranker
|
||||
|
||||
class Data_process():
|
||||
def __init__(self):
|
||||
self.chunk_size: int=1000
|
||||
self.chunk_overlap: int=100
|
||||
|
||||
def load_embedding_model(self, model_name='BAAI/bge-small-zh-v1.5', device='cpu', normalize_embeddings=True):
|
||||
"""
|
||||
加载嵌入模型。
|
||||
|
||||
参数:
|
||||
- model_name: 模型名称，字符串类型，默认为"BAAI/bge-small-zh-v1.5"。
|
||||
- device: 指定模型加载的设备，'cpu' 或 'cuda'，默认为'cpu'。
|
||||
- normalize_embeddings: 是否标准化嵌入向量，布尔类型，默认为 True。
|
||||
"""
|
||||
if not os.path.exists(embedding_path):
|
||||
os.makedirs(embedding_path, exist_ok=True)
|
||||
embedding_model_path = os.path.join(embedding_path,model_name.split('/')[1] + '.pkl')
|
||||
logger.info('Loading embedding model...')
|
||||
if os.path.exists(embedding_model_path):
|
||||
try:
|
||||
with open(embedding_model_path , 'rb') as f:
|
||||
embeddings = pickle.load(f)
|
||||
logger.info('Embedding model loaded.')
|
||||
return embeddings
|
||||
except Exception as e:
|
||||
logger.error(f'Failed to load embedding model from {embedding_model_path}')
|
||||
try:
|
||||
embeddings = HuggingFaceBgeEmbeddings(
|
||||
model_name=model_name,
|
||||
model_kwargs={'device': device},
|
||||
encode_kwargs={'normalize_embeddings': normalize_embeddings})
|
||||
logger.info('Embedding model loaded.')
|
||||
with open(embedding_model_path, 'wb') as file:
|
||||
pickle.dump(embeddings, file)
|
||||
except Exception as e:
|
||||
logger.error(f'Failed to load embedding model: {e}')
|
||||
return None
|
||||
return embeddings
|
||||
|
||||
def load_rerank_model(self, model_name='BAAI/bge-reranker-large'):
|
||||
"""
|
||||
加载重排名模型。
|
||||
|
||||
参数:
|
||||
- model_name (str): 模型的名称。默认为 'BAAI/bge-reranker-large'。
|
||||
|
||||
返回:
|
||||
- FlagReranker 实例。
|
||||
|
||||
异常:
|
||||
- ValueError: 如果模型名称不在批准的模型列表中。
|
||||
- Exception: 如果模型加载过程中发生任何其他错误。
|
||||
|
||||
"""
|
||||
if not os.path.exists(rerank_path):
|
||||
os.makedirs(rerank_path, exist_ok=True)
|
||||
rerank_model_path = os.path.join(rerank_path, model_name.split('/')[1] + '.pkl')
|
||||
logger.info('Loading rerank model...')
|
||||
if os.path.exists(rerank_model_path):
|
||||
try:
|
||||
with open(rerank_model_path , 'rb') as f:
|
||||
reranker_model = pickle.load(f)
|
||||
logger.info('Rerank model loaded.')
|
||||
return reranker_model
|
||||
except Exception as e:
|
||||
logger.error(f'Failed to load embedding model from {rerank_model_path}')
|
||||
try:
|
||||
reranker_model = FlagReranker(model_name, use_fp16=True)
|
||||
logger.info('Rerank model loaded.')
|
||||
with open(rerank_model_path, 'wb') as file:
|
||||
pickle.dump(reranker_model, file)
|
||||
except Exception as e:
|
||||
logger.error(f'Failed to load rerank model: {e}')
|
||||
raise
|
||||
|
||||
return reranker_model
|
||||
|
||||
|
||||
def extract_text_from_json(self, obj, content=None):
|
||||
"""
|
||||
抽取json中的文本，用于向量库构建
|
||||
|
||||
参数:
|
||||
- obj: dict,list,str
|
||||
- content: str
|
||||
|
||||
返回:
|
||||
- content: str
|
||||
"""
|
||||
if isinstance(obj, dict):
|
||||
for key, value in obj.items():
|
||||
try:
|
||||
content = self.extract_text_from_json(value, content)
|
||||
except Exception as e:
|
||||
print(f"Error processing value: {e}")
|
||||
elif isinstance(obj, list):
|
||||
for index, item in enumerate(obj):
|
||||
try:
|
||||
content = self.extract_text_from_json(item, content)
|
||||
except Exception as e:
|
||||
print(f"Error processing item: {e}")
|
||||
elif isinstance(obj, str):
|
||||
content += obj
|
||||
return content
|
||||
|
||||
|
||||
def split_document(self, data_path, chunk_size=500, chunk_overlap=100):
|
||||
"""
|
||||
切分data_path文件夹下的所有txt文件
|
||||
|
||||
参数:
|
||||
- data_path: str
|
||||
- chunk_size: int
|
||||
- chunk_overlap: int
|
||||
|
||||
返回：
|
||||
- split_docs: list
|
||||
"""
|
||||
|
||||
|
||||
# text_spliter = CharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)
|
||||
text_spliter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)
|
||||
split_docs = []
|
||||
logger.info(f'Loading txt files from {data_path}')
|
||||
if os.path.isdir(data_path):
|
||||
loader = DirectoryLoader(data_path, glob="**/*.txt",show_progress=True)
|
||||
docs = loader.load()
|
||||
split_docs = text_spliter.split_documents(docs)
|
||||
elif data_path.endswith('.txt'):
|
||||
file_path = data_path
|
||||
logger.info(f'splitting file {file_path}')
|
||||
text_loader = TextLoader(file_path, encoding='utf-8')
|
||||
text = text_loader.load()
|
||||
splits = text_spliter.split_documents(text)
|
||||
split_docs = splits
|
||||
logger.info(f'split_docs size {len(split_docs)}')
|
||||
return split_docs
|
||||
|
||||
|
||||
def split_conversation(self, path):
|
||||
"""
|
||||
按conversation块切分path文件夹下的所有json文件
|
||||
##TODO 限制序列长度
|
||||
"""
|
||||
# json_spliter = RecursiveJsonSplitter(max_chunk_size=500)
|
||||
logger.info(f'Loading json files from {path}')
|
||||
split_qa = []
|
||||
if os.path.isdir(path):
|
||||
# loader = DirectoryLoader(path, glob="**/*.json",show_progress=True)
|
||||
# jsons = loader.load()
|
||||
|
||||
for root, dirs, files in os.walk(path):
|
||||
for file in files:
|
||||
if file.endswith('.json'):
|
||||
file_path = os.path.join(root, file)
|
||||
logger.info(f'splitting file {file_path}')
|
||||
with open(file_path, 'r', encoding='utf-8') as f:
|
||||
data = json.load(f)
|
||||
# print(data)
|
||||
for conversation in data:
|
||||
# for dialog in conversation['conversation']:
|
||||
##按qa对切分,将每一轮qa转换为langchain_core.documents.base.Document
|
||||
# content = self.extract_text_from_json(dialog,'')
|
||||
# split_qa.append(Document(page_content = content))
|
||||
#按conversation块切分
|
||||
content = self.extract_text_from_json(conversation['conversation'], '')
|
||||
logger.info(f'content====={content}')
|
||||
split_qa.append(Document(page_content = content))
|
||||
# logger.info(f'split_qa size====={len(split_qa)}')
|
||||
return split_qa
|
||||
|
||||
|
||||
def load_knowledge(self, knowledge_pkl_path):
|
||||
'''
|
||||
读取或创建知识.pkl
|
||||
'''
|
||||
if not os.path.exists(knowledge_pkl_path):
|
||||
split_doc = self.split_document(doc_dir)
|
||||
split_qa = self.split_conversation(qa_dir)
|
||||
knowledge_chunks = split_doc + split_qa
|
||||
with open(knowledge_pkl_path, 'wb') as file:
|
||||
pickle.dump(knowledge_chunks, file)
|
||||
else:
|
||||
with open(knowledge_pkl_path , 'rb') as f:
|
||||
knowledge_chunks = pickle.load(f)
|
||||
return knowledge_chunks
|
||||
|
||||
|
||||
def create_vector_db(self, emb_model):
|
||||
'''
|
||||
创建并保存向量库
|
||||
'''
|
||||
logger.info(f'Creating index...')
|
||||
split_doc = self.split_document(doc_dir)
|
||||
split_qa = self.split_conversation(qa_dir)
|
||||
# logger.info(f'split_doc == {len(split_doc)}')
|
||||
# logger.info(f'split_qa == {len(split_qa)}')
|
||||
# logger.info(f'split_doc type == {type(split_doc[0])}')
|
||||
# logger.info(f'split_qa type== {type(split_qa[0])}')
|
||||
db = FAISS.from_documents(split_doc + split_qa, emb_model)
|
||||
db.save_local(vector_db_dir)
|
||||
return db
|
||||
|
||||
|
||||
def load_vector_db(self, knowledge_pkl_path=knowledge_pkl_path, doc_dir=doc_dir, qa_dir=qa_dir):
|
||||
'''
|
||||
读取向量库
|
||||
'''
|
||||
# current_os = platform.system()
|
||||
emb_model = self.load_embedding_model()
|
||||
if not os.path.exists(vector_db_dir) or not os.listdir(vector_db_dir):
|
||||
db = self.create_vector_db(emb_model)
|
||||
else:
|
||||
db = FAISS.load_local(vector_db_dir, emb_model, allow_dangerous_deserialization=True)
|
||||
return db
|
||||
|
||||
|
||||
def retrieve(self, query, vector_db, k=5):
|
||||
'''
|
||||
基于query对向量库进行检索
|
||||
'''
|
||||
retriever = vector_db.as_retriever(search_kwargs={"k": k})
|
||||
docs = retriever.invoke(query)
|
||||
return docs, retriever
|
||||
|
||||
##FlashrankRerank效果一般
|
||||
# def rerank(self, query, retriever):
|
||||
# compressor = FlashrankRerank()
|
||||
# compression_retriever = ContextualCompressionRetriever(base_compressor=compressor, base_retriever=retriever)
|
||||
# compressed_docs = compression_retriever.get_relevant_documents(query)
|
||||
# return compressed_docs
|
||||
|
||||
def rerank(self, query, docs):
|
||||
reranker = self.load_rerank_model()
|
||||
passages = []
|
||||
for doc in docs:
|
||||
passages.append(str(doc.page_content))
|
||||
scores = reranker.compute_score([[query, passage] for passage in passages])
|
||||
sorted_pairs = sorted(zip(passages, scores), key=lambda x: x[1], reverse=True)
|
||||
sorted_passages, sorted_scores = zip(*sorted_pairs)
|
||||
return sorted_passages, sorted_scores
|
||||
|
||||
|
||||
# def create_prompt(question, context):
|
||||
# from langchain.prompts import PromptTemplate
|
||||
# prompt_template = f"""请基于以下内容回答问题：
|
||||
|
||||
# {context}
|
||||
|
||||
# 问题: {question}
|
||||
# 回答:"""
|
||||
# prompt = PromptTemplate(
|
||||
# template=prompt_template, input_variables=["context", "question"]
|
||||
# )
|
||||
# logger.info(f'Prompt: {prompt}')
|
||||
# return prompt
|
||||
|
||||
def create_prompt(question, context):
|
||||
prompt = f"""请基于以下内容: {context} 给出问题答案。问题如下: {question}。回答:"""
|
||||
logger.info(f'Prompt: {prompt}')
|
||||
return prompt
|
||||
|
||||
def test_zhipu(prompt):
|
||||
from zhipuai import ZhipuAI
|
||||
api_key = "" # 填写您自己的APIKey
|
||||
if api_key == "":
|
||||
raise ValueError("请填写api_key")
|
||||
client = ZhipuAI(api_key=api_key)
|
||||
response = client.chat.completions.create(
|
||||
model="glm-4", # 填写需要调用的模型名称
|
||||
messages=[
|
||||
{"role": "user", "content": prompt[:100]}
|
||||
],
|
||||
)
|
||||
print(response.choices[0].message)
|
||||
|
||||
if __name__ == "__main__":
|
||||
logger.info(data_dir)
|
||||
if not os.path.exists(data_dir):
|
||||
os.mkdir(data_dir)
|
||||
dp = Data_process()
|
||||
# faiss_index, knowledge_chunks = dp.load_index_and_knowledge(knowledge_pkl_path='')
|
||||
vector_db = dp.load_vector_db()
|
||||
# 按照query进行查询
|
||||
# query = "儿童心理学说明-内容提要-目录 《儿童心理学》1993年修订版说明 《儿童心理学》是1961年初全国高等学校文科教材会议指定朱智贤教授编 写的。1962年初版，1979年再版。"
|
||||
# query = "我现在处于高三阶段，感到非常迷茫和害怕。我觉得自己从出生以来就是多余的，没有必要存在于这个世界。无论是在家庭、学校、朋友还是老师面前，我都感到被否定。我非常难过，对高考充满期望但成绩却不理想，我现在感到非常孤独、累和迷茫。您能给我提供一些建议吗？"
|
||||
# query = "这在一定程度上限制了其思维能力，特别是辩证 逻辑思维能力的发展。随着年龄的增长，初中三年级学生逐步克服了依赖性"
|
||||
# query = "我现在处于高三阶段，感到非常迷茫和害怕。我觉得自己从出生以来就是多余的，没有必要存在于这个世界。无论是在家庭、学校、朋友还是老师面前，我都感到被否定。我非常难过，对高考充满期望但成绩却不理想"
|
||||
# query = "我现在心情非常差，有什么解决办法吗？"
|
||||
query = "我最近总感觉胸口很闷，但医生检查过说身体没问题。可我就是觉得喘不过气来，尤其是看到那些旧照片，想起过去的日子"
|
||||
docs, retriever = dp.retrieve(query, vector_db, k=10)
|
||||
logger.info(f'Query: {query}')
|
||||
logger.info("Retrieve results:")
|
||||
for i, doc in enumerate(docs):
|
||||
logger.info(str(i) + '\n')
|
||||
logger.info(doc)
|
||||
# print(f'get num of docs:{len(docs)}')
|
||||
# print(docs)
|
||||
passages,scores = dp.rerank(query, docs)
|
||||
logger.info("After reranking...")
|
||||
for i in range(len(scores)):
|
||||
logger.info(str(scores[i]) + '\n')
|
||||
logger.info(passages[i])
|
||||
prompt = create_prompt(query, passages[0])
|
||||
test_zhipu(prompt) ## 如果显示'Server disconnected without sending a response.'可能是由于上下文窗口限制
|
||||
247
scripts/README_Model_Uploading.md
Normal file
247
scripts/README_Model_Uploading.md
Normal file
@ -0,0 +1,247 @@
|
||||
# 模型上传指南
|
||||
|
||||
## OpenXLab浦源平台
|
||||
|
||||
### OpenXLab平台介绍
|
||||
|
||||
<div align="center">
|
||||
<img src="./asserts/openxlab.png" width="600"/>
|
||||
<div align="center">
|
||||
</div>
|
||||
</div>
|
||||
|
||||
OpenXLab浦源 内容平台 是面向 AI 研究员和开发者提供 AI 领域的一站式服务平台，包含数据集中心、模型中心和应用中心。内容平台为 AI 研究员和开发者提供了所需的模型训练物料，同时也为他们提供了模型推理应用的托管服务。此外，内容平台致力于打造一个 AI 数据集、模型与应用的交流社区，为 AI 研究者提供一个分享和交流的平台。通过内容平台，AI 研究者可以更好地展示自己的模型能力，并激发创造力，助力 AI 生态的可持续发展。
|
||||
|
||||
更多介绍请查看[OpenXLab浦源平台介绍](https://openxlab.org.cn/docs/intro.html)
|
||||
|
||||
<!-- 应用中心：应用中心提供应用托管的服务，用户只需遵循平台规范，通过简单的前端封装组件（Gradio）即可构建模型推理应用演示 demo，应用中心提供免费应用部署的能力，普通用户也可在应用中心中交互式体验模型的能力，更好帮助用户寻找想要的学术模型或应用服务。通过前端封装组件和平台的 SDK 工具，帮助 AI 开发者简单快速构建人工智能应用。
|
||||
|
||||
模型中心：支持丰富模型管理方式，模型中心基于模型元信息标准规范，支持用户上传、存储、检索、评测各类模型。基于平台内的命令行工具，便于 AI 开发者上传和发布模型文件，搭建对象存储体系，提供大文件存储能力，快速上传下载功能，便于 AI 开发者进行模型存储。
|
||||
|
||||
数据集中心：支持多元数据管理，数据中心提供公开数据集的展示、检索和下载等，同时提供私有数据集的上传、管理和发布功能，支持用户自建数据集的开放共享。数据集中心为人工智能研究者提供免费开源的数据集，通过数据集中心，研究者可以获得格式统一的各领域经典数据集。通过平台的搜索功能，研究者可以迅速便捷地找到自己所需数据集；通过平台的统一格式，研究者可以便捷地对跨数据集任务进行开发。
|
||||
|
||||
<div align="center">
|
||||
<img src="./asserts/平台概述.e6d980f8.png" width="600"/>
|
||||
<div align="center">
|
||||
</div>
|
||||
</div>
|
||||
|
||||
内容平台中，模型仓库存储模型相关的权重文件，应用仓库管理部署应用，为了简化贡献者的维护成本，模型相关的算法训练代码和应用的相关代码托管至 GitHub 中，即 GitHub 中可以存放算法训练的代码和应用相关的代码，贡献者只需维护 GitHub 仓库代码即可，无需多方维护代码，内容平台只提供模型权重的存储服务和应用的部署服务。
|
||||
|
||||
<div align="center">
|
||||
<img src="./asserts/GitHub与平台的关系.bee7809e.png" width="600"/>
|
||||
<div align="center">
|
||||
</div>
|
||||
</div> -->
|
||||
|
||||
## 模型创建流程
|
||||
|
||||
### 要点强调
|
||||
|
||||
- 浦源-模型中心提供目前支持通过Git命令进行文件上传
|
||||
- 使用该方法进行文件上传前，请您确认已安装Git
|
||||
- 由于上传需要进行权限校验，这里我们推荐使用VSCode远程ssh连接InternLM AI Studio, 获取XLab秘钥
|
||||
|
||||
### 创建具体步骤
|
||||
|
||||
- 步骤1：点击“创建模型”按钮
|
||||
- 步骤2：填写仓库相关信息
|
||||
- **步骤3：上传模型相关文件**
|
||||
|
||||
更多详情和操作步骤请查看, 请参考[**模型创建流程 **(步骤1和2)](https://openxlab.org.cn/docs/models/%E6%A8%A1%E5%9E%8B%E5%88%9B%E5%BB%BA%E6%B5%81%E7%A8%8B.html)和[**上传模型**(步骤3)](https://openxlab.org.cn/docs/models/%E4%B8%8A%E4%BC%A0%E6%A8%A1%E5%9E%8B.html), 这里我们将给出所用到的基本步骤和需要注意的操作要点.
|
||||
|
||||
## 上传模型
|
||||
|
||||
### 上传具体步骤
|
||||
|
||||
- **步骤1：安装git lfs**
|
||||
- **步骤2：配置git和lfs**
|
||||
- **步骤3：配置OpenXLab秘钥**
|
||||
- 步骤4：在本地的文件夹内调整文件
|
||||
- 步骤5：上传本地文件夹中的模型文件到OpenXLab
|
||||
- 步骤6：上传后查看和添加README.md
|
||||
|
||||
这里展示最顺利的截图
|
||||
|
||||
<div align="center">
|
||||
<img src="./asserts/full_upload.png" width="600"/>
|
||||
<div align="center">
|
||||
</div>
|
||||
</div>
|
||||
|
||||
### 1. 安装git lfs
|
||||
|
||||
```bash
|
||||
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh
|
||||
apt install git-lfs
|
||||
```
|
||||
|
||||
### 2. 配置git和lfs
|
||||
|
||||
```bash
|
||||
git lfs install # 这个很关键
|
||||
git clone https://code.openxlab.org.cn//chg0901/EmoLLM-InternLM7B-base.git # 要上传的模型链接, 由步骤1和2创建
|
||||
```
|
||||
|
||||
`git lfs install`会出现一个Error，请忽略，这是因为这条命令执行的
|
||||
|
||||
### 3. 配置OpenXLab秘钥
|
||||
|
||||
- 详情请参考[**密钥管理**](https://openxlab.org.cn/security?tab=git), 获取您的 Git Access Token
|
||||
- 点击 “**添加令牌**” 按钮
|
||||
- 由于后续需要进行文件上传，所以请您在新建token时，选择 **“可写” 权限**
|
||||
- **注意：**最好是**重新创建**一个**新的令牌**，*旧的令牌*可能会导致***上传权限失败***
|
||||
|
||||
<div align="center">
|
||||
<img src="./asserts/unautheorized.png" width="600"/>
|
||||
<div align="center">
|
||||
</div>
|
||||
</div>
|
||||
|
||||
### 4. 在本地的文件夹内调整文件（文件夹名同仓库同名）和配置Git信息
|
||||
|
||||
将merge后的模型文件复制到git clone后的文件夹中
|
||||
|
||||
```bash
|
||||
cd ./merge # merge目录下为合并后的模型
|
||||
cp ./* /root/EmoLLM-InternLM7B-base/ # 复制模型到clone后的文件夹
|
||||
|
||||
# 配置Git信息
|
||||
git config --global user.email "your email"
|
||||
git config --global user.name "your OpenXLab id" # OpenXLab id
|
||||
git config --global user.password "your new key" # 新的OpenXLab秘钥
|
||||
|
||||
```
|
||||
|
||||
### 5. 上传本地文件夹中的模型文件到OpenXLab
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "commit EmoLLM-InternLM7B-base"
|
||||
git push
|
||||
```
|
||||
|
||||
push的时候, 需要填写username和password三次,
|
||||
|
||||
<div align="center">
|
||||
<img src="./asserts/username_password.png" width="600"/>
|
||||
<div align="center">
|
||||
</div>
|
||||
</div>
|
||||
|
||||
### 6. 上传后查看和添加README.md
|
||||
|
||||
上传完模型, 还可以复制之前上传的`README.md`文件到自己的仓库中.
|
||||
|
||||
处理完之后, 就可以看到自己的模型Repo了.
|
||||
|
||||
<div align="center">
|
||||
<img src="./asserts/result1.png" width="600"/>
|
||||
<div align="center">
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div align="center">
|
||||
<img src="./asserts/result2.png" width="600"/>
|
||||
<div align="center">
|
||||
</div>
|
||||
</div>
|
||||
|
||||
### 可能遇到的问题
|
||||
|
||||
可以查看下面的截图, 查看bug和解决方法以及所用的bash命令.
|
||||
|
||||
出现这个问题的原因是因为上传不成功或者上传被打断.
|
||||
|
||||
<div align="center">
|
||||
<img src="./asserts/upload_error.png" width="600"/>
|
||||
<div align="center">
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div align="center">
|
||||
<img src="./asserts/upload_error_solution.png" width="600"/>
|
||||
<div align="center">
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div align="center">
|
||||
<img src="./asserts/upload_error_solution2.png" width="600"/>
|
||||
<div align="center">
|
||||
</div>
|
||||
</div>
|
||||
|
||||
bash命令如下:
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "commit EmoLLM-InternLM7B-base"
|
||||
git push # 出现error
|
||||
|
||||
# solution1
|
||||
git gc --prune=now
|
||||
git remote prune origin
|
||||
git push
|
||||
|
||||
# solution2 (可能solution1无效)
|
||||
git update-ref -d refs/heads/main
|
||||
git fetch
|
||||
git merge origin/main
|
||||
|
||||
# error 解决, 重新上传
|
||||
git push
|
||||
git commit -m "commit EmoLLM-InternLM7B-base"
|
||||
git push
|
||||
```
|
||||
|
||||
## ModelScope
|
||||
|
||||
### ModelScope平台介绍
|
||||
|
||||
ModelScope旨在打造下一代开源的模型即服务共享平台，为泛AI开发者提供灵活、易用、低成本的一站式模型服务产品，让模型应用更简单！
|
||||
|
||||
我们希望在汇集行业领先的预训练模型，减少开发者的重复研发成本，提供更加绿色环保、开源开放的AI开发环境和模型服务，助力绿色“数字经济”事业的建设。
|
||||
ModelScope平台将以开源的方式提供多类优质模型，开发者可在平台上免费体验与下载使用。
|
||||
|
||||
### 模型创建
|
||||
|
||||
ModelScope平台内的模型创建和OpenXLab, 这里不再赘述, 可以点击[ModelScope模型创建链接地址](https://modelscope.cn/models/create)自行填写.
|
||||
|
||||
<div align="center">
|
||||
<img src="./asserts/ms_create.png" width="600"/>
|
||||
<div align="center">
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div align="center">
|
||||
<img src="./asserts/ms_config.png" width="600"/>
|
||||
<div align="center">
|
||||
</div>
|
||||
</div>
|
||||
|
||||
### 使用Python SDK上传模型
|
||||
|
||||
可以使用modelscope modelhub来将已经训练好的模型上传到ModelScope平台,
|
||||
|
||||
ModelScope的上传比OpenXLab简单不少, 在ModelScope社区网页创建对应模型之后，只需要**配置访问令牌(请从ModelScope`个人中心->访问令牌获取`)**, 然后将本地模型目录通过push_model接口进行上传即可.
|
||||
|
||||
需要注意的是, **ModelScope要求上传的模型目录含有`configuration.json`文件**, 我们训练的merge模型目录只有`config.json`, 因此可以复制这个文件然后修改文件名即可.
|
||||
|
||||
```python
|
||||
from modelscope.hub.api import HubApi
|
||||
|
||||
YOUR_ACCESS_TOKEN = '请从ModelScope个人中心->访问令牌获取'
|
||||
|
||||
api = HubApi()
|
||||
api.login(YOUR_ACCESS_TOKEN)
|
||||
api.push_model(
|
||||
model_id="yourname/your_model_id",
|
||||
model_dir="my_model_dir" # 本地模型目录，要求目录中必须包含configuration.json
|
||||
)
|
||||
```
|
||||
|
||||
<div align="center">
|
||||
<img src="./asserts/ms_upload.png" width="900"/>
|
||||
<div align="center">
|
||||
</div>
|
||||
</div>
|
||||
BIN
scripts/asserts/GitHub与平台的关系.bee7809e.png
Normal file
BIN
scripts/asserts/GitHub与平台的关系.bee7809e.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 54 KiB |
BIN
scripts/asserts/full_upload.png
Normal file
BIN
scripts/asserts/full_upload.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 408 KiB |
BIN
scripts/asserts/ms_config.png
Normal file
BIN
scripts/asserts/ms_config.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 632 KiB |
BIN
scripts/asserts/ms_create.png
Normal file
BIN
scripts/asserts/ms_create.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 817 KiB |
BIN
scripts/asserts/ms_upload.png
Normal file
BIN
scripts/asserts/ms_upload.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 98 KiB |
BIN
scripts/asserts/openxlab.png
Normal file
BIN
scripts/asserts/openxlab.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 172 KiB |
BIN
scripts/asserts/result1.png
Normal file
BIN
scripts/asserts/result1.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 212 KiB |
BIN
scripts/asserts/result2.png
Normal file
BIN
scripts/asserts/result2.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 181 KiB |
BIN
scripts/asserts/unautheorized.png
Normal file
BIN
scripts/asserts/unautheorized.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 66 KiB |
BIN
scripts/asserts/upload_error.png
Normal file
BIN
scripts/asserts/upload_error.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 80 KiB |
BIN
scripts/asserts/upload_error_solution.png
Normal file
BIN
scripts/asserts/upload_error_solution.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 80 KiB |
BIN
scripts/asserts/upload_error_solution2.png
Normal file
BIN
scripts/asserts/upload_error_solution2.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 63 KiB |
BIN
scripts/asserts/username_password.png
Normal file
BIN
scripts/asserts/username_password.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 108 KiB |
BIN
scripts/asserts/平台概述.e6d980f8.png
Normal file
BIN
scripts/asserts/平台概述.e6d980f8.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 50 KiB |
11
scripts/upload_modelscope.py
Normal file
11
scripts/upload_modelscope.py
Normal file
@ -0,0 +1,11 @@
|
||||
|
||||
from modelscope.hub.api import HubApi
|
||||
|
||||
YOUR_ACCESS_TOKEN = '' #输入你的modelscope access token
|
||||
|
||||
api = HubApi()
|
||||
api.login(YOUR_ACCESS_TOKEN)
|
||||
api.push_model(
|
||||
model_id="zealot5209/EmoLLM-Scientist", #your_name/model_id
|
||||
model_dir="./merged" # 本地模型目录，要求目录中必须包含configuration.json
|
||||
)
|
||||
192
xtuner_config/README_internlm2_7b_base_qlora.md
Normal file
192
xtuner_config/README_internlm2_7b_base_qlora.md
Normal file
@ -0,0 +1,192 @@
|
||||
# InternLM2 7B Base QLoRA 微调指南
|
||||
|
||||
## 模型基座与配置文件
|
||||
|
||||
- 本项目在[**internlm2_7b_chat_qlora_e3**模型](./internlm2_7b_chat_qlora_e3.py)微调[指南](./README.md)的基础上，更新了对[**internlm2_7b_base_qlora_e3（配置文件）**](./internlm2_7b_base_qlora_e10_M_1e4_32_64.py)**模型**的微调。
|
||||
|
||||
## 数据
|
||||
|
||||
### 数据集
|
||||
|
||||
只采用了通用的数据集，不包括带有Role-Play色彩的数据集，详情请查看[数据集](../datasets/README.md)页面
|
||||
|
||||
| Category | Dataset | Type | Total |
|
||||
| :---------: | :-------------------: | :----------: | :-----: |
|
||||
| General | data | Conversation | 5600+ |
|
||||
| General | data_pro | Conversation | 36500+ |
|
||||
| General | multi_turn_dataset_1 | Conversation | 36,000+ |
|
||||
| General | multi_turn_dataset_2 | Conversation | 27,000+ |
|
||||
| General | single_turn_dataset_1 | QA | 14000+ |
|
||||
| General | single_turn_dataset_2 | QA | 18300+ |
|
||||
|
||||
### 数据集处理
|
||||
|
||||
#### 数据格式
|
||||
|
||||
训练用的数据，被处理成单轮或者多轮对话的形式，如
|
||||
|
||||
```python
|
||||
[
|
||||
# 多轮对话
|
||||
{
|
||||
"conversation": [
|
||||
{
|
||||
"input": "我最近很迷茫，高三后的压力让我感到恐惧，我对自己的存在感到怀疑，您能不能给我一些建议呢？",
|
||||
"output": "看到你面临的困境与压力，真心希望能够帮助你。首先，我想说的是，你并不是孤独的，高考前的困惑与迷茫是很常见的。不要给自己太大的压力，给自己短暂的放松。你认识到问题所在，这已经是一个很大的进步了。",
|
||||
"system": "你是心理健康助手EmoLLM，由EmoLLM团队打造。你旨在通过专业心理咨询，协助来访者完成心理诊断。请充分利用专业心理学知识与咨询技术，一步步帮助来访者解决心理问题。"
|
||||
},
|
||||
{
|
||||
"input": "谢谢您的鼓励，但我觉得自己学习成绩不好，没有进步，这让我更加困惑。",
|
||||
"output": "学习成绩好坏不是解决问题的关键，专注于自己的进步更为重要。为了达到自己的目标，制定一份合理的计划，按部就班的完成每一项任务。有时候自学不如求助他人，不要害怕问老师和同学问题，以便获得更好的帮助和理解。"
|
||||
},
|
||||
{
|
||||
"input": "我认为我缺乏自信，不知道怎样扭转自己的消极情绪，您能给我一些建议吗？",
|
||||
"output": "建立自信需要时间和努力。让我们从一点一滴的进步中建立自信心。设立一些具体的目标，完成这些目标将会培养自己的自信。鼓励自己，在每天结束前，总结一下自己的成就，这将帮助你更好的看到自身的优点。"
|
||||
},
|
||||
{
|
||||
"input": "我感到很无助，不知道该如何调整自己的心态。",
|
||||
"output": "了解自己的情绪并寻找有效的调节方法是处理感到无助情况的关键。比如爱好运动、听音乐和阅读书籍等，这些活动对于您的心理健康都是有积极帮助的。此外，在处理感到无助的情况下，家庭和朋友的支持也是非常关键的。"
|
||||
},
|
||||
{
|
||||
"input": "我感觉我的思维和学习方法都很糟糕，该怎样改进呢？",
|
||||
"output": "学习方法确实很重要，它会对你的成绩和情绪有很大的影响。最有效的学习方法是利用时间来做练习，多做一些题目，练习自己的掌握能力。建立起对知识的兴趣，这需要不断的积累，不要害怕失败或挫折，它们会成为助你成长的宝贵财富。"
|
||||
},
|
||||
{
|
||||
"input": "我会按照您的建议努力，但想问一下，我该如何面对可能的失败？",
|
||||
"output": "每个人都会面对失败和挫折，不要让它们打败你。碰到失败时，重新审视自己，分析分析自己的不足，寻找方法去改进。记得勇敢地看向未来，不要让过去的失败阻碍了自己前进的脚步。"
|
||||
},
|
||||
{
|
||||
"input": "非常感谢您的建议和支持，我会努力去实践。",
|
||||
"output": "这是我应该做的，你有勇气面对自己困难，你并不孤独，我们会一起为你打气，祝愿你早日获得心灵的平静和所有的成功。"
|
||||
}
|
||||
]
|
||||
},
|
||||
# 单轮对话
|
||||
{
|
||||
"conversation": [
|
||||
{
|
||||
"system": "你是心理健康助手EmoLLM，由EmoLLM团队打造。你旨在通过专业心理咨询，协助来访者完成心理诊断。请充分利用专业心理学知识与咨询技术，一步步帮助来访者解决心理问题。",
|
||||
"input": "一想到事情就紧张……?\n我最近因为考试，觉得心情很不好，自己没有及格，而且，明明觉得自己比室友付出的多，努力的多，平时做题也是一样的，可是不知道为什么只有我不及格，感觉很苦恼。而且我只要一有什么事情就十分紧张，考试也是，心跳加速，手心出汗，而且大脑一片空白，有时甚至会头疼，高考以及现在的考试都是这样。而且我觉得一情绪激动的时候就会咳嗽，想哭哭不出来，感觉很难受，头也会疼。就是觉得心情不好，但是有不敢表现出来。我该怎么办呢？",
|
||||
"output": "为什么不敢表现出来呢？试试表现出来，会怎样？\n有时候我们越抗拒，就会越强化。当我们接受了，反而症状就消失了。\n那我们要怎样接受呢？\n这本书《心灵的七种兵器》推荐看看哦，内容精简，又有真实事例可以帮助理解。\n我们为什么改不掉这个所谓的坏习惯。\n"
|
||||
}
|
||||
]
|
||||
},
|
||||
```
|
||||
|
||||
### 数据处理
|
||||
|
||||
- 使用 `../datasets/process.py` 以处理 **multi_turn_dataset(1 和 2，QA数据转单轮对话)**， `data.json` 和 `data_pro.json` 文件（两个多轮对话），以添加或者调整 **`system` prompt**
|
||||
- 使用 `../datasets/processed/process_single_turn_conversation_construction.py` 处理 **single-turn dataset** (1 和 2)，修改 (`input` 和 `ouput`) ，并在每次 **conversation** 中添加 **`system` prompt**
|
||||
- 使用 `../datasets/processed/process_merge.py` 用于合并 `../datasets/processed/` 目录下**6个更新后的数据集**，生成一个合并后的数据集 `combined_data.json`用于最终训练
|
||||
|
||||
### 数据量与训练epochs设置
|
||||
|
||||
- 由于采用了更大的数据集，我们对模型进行了**10 epoch**的训练，读者可以根据训练过程中的输出和loss变化，进行训练的终止和模型的挑选，也可以采用更加专业的评估方法，来对模型评测。
|
||||
- 在我们公布的托管于OpenXlab微调后的 internlm2_7b_chat_qlora微调模型中，我们保留了两个版本，一个是[5 epoch模型](https://openxlab.org.cn/models/detail/chg0901/EmoLLM-InternLM7B-base/tree/main)，另一个是[10 epoch模型](https://openxlab.org.cn/models/detail/chg0901/EmoLLM-InternLM7B-base-10e/tree/main)版本（**ModelScope**模型：[5 epoch模型](https://www.modelscope.cn/models/chg0901/EmoLLM-InternLM7B-base/files)和[10 epoch模型](https://www.modelscope.cn/models/chg0901/EmoLLM-InternLM7B-base-10e/files)）。
|
||||
|
||||
## 基于XTuner的微调🎉🎉🎉🎉🎉
|
||||
|
||||
### 环境准备
|
||||
|
||||
```markdown
|
||||
datasets==2.16.1
|
||||
deepspeed==0.13.1
|
||||
einops==0.7.0
|
||||
flash_attn==2.5.0
|
||||
openxlab==0.0.34
|
||||
peft==0.7.1
|
||||
sentencepiece==0.1.99
|
||||
torch==2.1.2
|
||||
transformers==4.36.2
|
||||
mmengine==0.10.3
|
||||
xtuner==0.1.15
|
||||
flash_attn==2.5.0
|
||||
```
|
||||
|
||||
也可以一键安装
|
||||
|
||||
```bash
|
||||
cd xtuner_config/
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
温馨提示：flash_attn的安装可能需要在本地编译，大约需要一到两小时，可以去[flash-attention](https://github.com/Dao-AILab/flash-attention/releases)中，查找和自己机器配置匹配的whl安装包或者采用InternLM AI studio提供的2.4.2版本whl安装包，自行安装，如：
|
||||
|
||||
```bash
|
||||
# from flash-attention
|
||||
pip install flash_attn-2.5.0+cu122torch2.1cxx11abiTRUE-cp310-cp310-linux_x86_64.whl
|
||||
|
||||
# from InternLM AI studio share folder
|
||||
pip install /root/share/wheels/flash_attn-2.4.2+cu118torch2.0cxx11abiTRUE-cp310-cp310-linux_x86_64.whl
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 微调
|
||||
|
||||
```bash
|
||||
cd xtuner_config/
|
||||
xtuner train internlm2_7b_base_qlora_e10_M_1e4_32_64.py --deepspeed deepspeed_zero2
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 将得到的 PTH 模型转换为 HuggingFace 模型
|
||||
|
||||
**即：生成 Adapter 文件夹**
|
||||
|
||||
```bash
|
||||
cd xtuner_config/
|
||||
mkdir hf
|
||||
export MKL_SERVICE_FORCE_INTEL=1
|
||||
|
||||
xtuner convert pth_to_hf internlm2_7b_base_qlora_e10_M_1e4_32_64.py ./work_dirs/internlm2_7b_base_qlora_e10_M_1e4_32_64/epoch_5.pth ./hf
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 将 HuggingFace adapter 合并到大语言模型
|
||||
|
||||
```bash
|
||||
xtuner convert merge /root/share/model_repos/internlm2-base-7b ./hf ./merged --max-shard-size 2GB
|
||||
# xtuner convert merge \
|
||||
# ${NAME_OR_PATH_TO_LLM} \
|
||||
# ${NAME_OR_PATH_TO_ADAPTER} \
|
||||
# ${SAVE_PATH} \
|
||||
# --max-shard-size 2GB
|
||||
```
|
||||
|
||||
### 10 epoch 模型的处理
|
||||
|
||||
```bash
|
||||
|
||||
cd xtuner_config/
|
||||
mkdir hf10
|
||||
export MKL_SERVICE_FORCE_INTEL=1
|
||||
|
||||
xtuner convert pth_to_hf internlm2_7b_base_qlora_e10_M_1e4_32_64.py ./work_dirs/internlm2_7b_base_qlora_e10_M_1e4_32_64/epoch_10.pth ./hf
|
||||
|
||||
xtuner convert merge /root/share/model_repos/internlm2-base-7b ./hf10 ./merged10 --max-shard-size 2GB
|
||||
# xtuner convert merge \
|
||||
# ${NAME_OR_PATH_TO_LLM} \
|
||||
# ${NAME_OR_PATH_TO_ADAPTER} \
|
||||
# ${SAVE_PATH} \
|
||||
# --max-shard-size 2GB
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 测试
|
||||
|
||||
```bash
|
||||
cd demo/
|
||||
python cli_internlm2.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 其他
|
||||
|
||||
欢迎大家给[xtuner](https://github.com/InternLM/xtuner)和[EmoLLM](https://github.com/aJupyter/EmoLLM)点点star~
|
||||
|
||||
🎉🎉🎉🎉🎉
|
||||
91
xtuner_config/README_scientist.md
Normal file
91
xtuner_config/README_scientist.md
Normal file
@ -0,0 +1,91 @@
|
||||
# EmoLLM_Scientist微调指南
|
||||
[EmoLLM_Scientist模型地址]https://modelscope.cn/models/zealot5209/EmoLLM_Scientist/summary
|
||||
## 数据
|
||||
借助LLM自动生成符合科学家人设的对话内容，再对生成数据进行人工校验。最终微调数据包含约3800段对话，每段对话包含5~10轮。
|
||||
#### 数据路径：
|
||||
`datasets\scientist.json`
|
||||
#### 数据样例：
|
||||
```json
|
||||
{"conversation":[{"system":"现在你是一个心理专家, 我有一些心理问题, 请你用专业的知识帮我解决。",
|
||||
"input":"我最近对学习特别兴奋，每天可以熬夜到凌晨三点，我觉得这样效率很高，但室友说我有点过度了，我不这么认为。\n\n",
|
||||
"output":"很高兴看到你对学业充满热情，你现在的学习状态让我想到了心理学上的“过度激活”现象。能否详细描述一下你为何选择深夜学习，以及这样做的实际效果如何呢？\n\n"},
|
||||
{"input":"我觉得晚上思路更清晰，而且能集中精力完成很多任务。确实白天上课时有点困倦，但我总觉得自己在高效利用时间。\n\n",
|
||||
"output":"理解你的想法，不过科学研究显示，长期熬夜可能影响生物钟，进而导致认知功能下降。作为你的科学家心理咨询师，我想知道，你是否注意到这种高强度学习方式对日常生活其他方面的潜在影响，比如社交活动或身体健康？\n\n"},
|
||||
{"input":"确实，我发现自己减少了锻炼和跟朋友相处的时间，但我觉得这些都是暂时牺牲，为了更好的成绩。\n\n",
|
||||
"output":"你这种短期目标优先的策略值得肯定，但从长远看，身心健康与社会交往同样重要。有没有想过制定一个兼顾学习与生活的合理计划？我们可以一起探讨如何优化时间管理，确保高效学习的同时，也不忽视生活质量。\n\n"},
|
||||
{"input":"我也想平衡，只是不知道怎么做才好。\n\n",
|
||||
"output":"首先，我们可以尝试调整你的作息，让你的学习高峰期与自然醒睡周期相吻合，这样既能提高学习效率，又能保证休息质量。其次，我们可以设定每日学习目标和休闲时间，科学分配任务。最后，记得适时奖励自己，庆祝每个小成就，这样可以激励持续进步。你觉得这样的方案是否可行？\n\n"},
|
||||
{"input":"听起来不错，我会试试看的，但怕自己坚持不下来。\n\n",
|
||||
"output":"理解你的担忧，改变习惯需要时间和耐心。我们可以通过行为科学的方法逐步调整，如采用“番茄工作法”，结合正向强化策略，帮助你建立新的学习习惯。记住，每一次微小的进步都值得庆祝，我会陪伴并支持你的改变过程。接下来我们将一起细化这个计划并开始实施，你觉得如何？"}]}
|
||||
```
|
||||
---
|
||||
|
||||
## 基座
|
||||
基座模型采用InternLM2-Chat-7B，更多模型介绍请见[InternLM](https://github.com/InternLM/InternLM)
|
||||
|
||||
## 训练方式
|
||||
基于[xtuner](https://github.com/InternLM/xtuner)进行微调。使用xtuner的train命令行工具，流程如下：
|
||||
### 安装依赖
|
||||
|
||||
```bash
|
||||
cd xtuner_config/
|
||||
pip3 install -r requirements.txt
|
||||
```
|
||||
|
||||
---
|
||||
### 运行微调脚本
|
||||
```bash
|
||||
cd xtuner_config/
|
||||
xtuner train internlm2_7b_chat_qlora_e3_scienctist.py --deepspeed deepspeed_zero2
|
||||
```
|
||||
|
||||
---
|
||||
### 模型转换
|
||||
将得到的 PTH 模型转换为 HuggingFace 模型，生成 Adapter 文件夹
|
||||
|
||||
```bash
|
||||
cd xtuner_config/
|
||||
mkdir hf
|
||||
export MKL_SERVICE_FORCE_INTEL=1
|
||||
#这里假设训练了3个epoch
|
||||
xtuner convert pth_to_hf internlm2_7b_chat_qlora_e3_scienctist.py ./work_dirs/internlm2_7b_chat_qlora_e3_scienctist/epoch_3.pth ./hf
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 模型合并
|
||||
将 HuggingFace adapter 合并到大语言模型
|
||||
```bash
|
||||
xtuner convert merge ./internlm2-chat-7b ./hf ./merged --max-shard-size 2GB
|
||||
# xtuner convert merge \
|
||||
# ${NAME_OR_PATH_TO_LLM} \
|
||||
# ${NAME_OR_PATH_TO_ADAPTER} \
|
||||
# ${SAVE_PATH} \
|
||||
# --max-shard-size 2GB
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 测试
|
||||
|
||||
```
|
||||
cd demo/
|
||||
python cli_internlm2_scientist.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 模型上传
|
||||
完成测试后可将模型上传到ModelScope和Openxlab平台(不建议在Windows下操作)
|
||||
#### ModelScope
|
||||
[Openxlab模型上传](https://openxlab.org.cn/docs/models/%E4%B8%8A%E4%BC%A0%E6%A8%A1%E5%9E%8B.html)
|
||||
脚本：`scripts/upload_modelscope.py`
|
||||
|
||||
#### Openxlab
|
||||
[ModelScope模型上传](https://modelscope.cn/docs/%E6%A8%A1%E5%9E%8B%E7%9A%84%E5%88%9B%E5%BB%BA%E4%B8%8E%E6%96%87%E4%BB%B6%E4%B8%8A%E4%BC%A0)
|
||||
|
||||
## 其他
|
||||
|
||||
欢迎大家给[xtuner](https://github.com/InternLM/xtuner)和[EmoLLM](https://github.com/aJupyter/EmoLLM)点点star~
|
||||
|
||||
🎉🎉🎉🎉🎉
|
||||
210
xtuner_config/internlm2_7b_base_qlora_e10_M_1e4_32_64.py
Normal file
210
xtuner_config/internlm2_7b_base_qlora_e10_M_1e4_32_64.py
Normal file
@ -0,0 +1,210 @@
|
||||
# Copyright (c) OpenMMLab. All rights reserved.
|
||||
import torch
|
||||
from datasets import load_dataset
|
||||
from mmengine.dataset import DefaultSampler
|
||||
from mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,
|
||||
LoggerHook, ParamSchedulerHook)
|
||||
from mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR
|
||||
from peft import LoraConfig
|
||||
from torch.optim import AdamW
|
||||
from transformers import (AutoModelForCausalLM, AutoTokenizer,
|
||||
BitsAndBytesConfig)
|
||||
|
||||
from xtuner.dataset import process_hf_dataset
|
||||
from xtuner.dataset.collate_fns import default_collate_fn
|
||||
from xtuner.dataset.map_fns import template_map_fn_factory
|
||||
from xtuner.engine import DatasetInfoHook, EvaluateChatHook
|
||||
from xtuner.model import SupervisedFinetune
|
||||
from xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE
|
||||
|
||||
#######################################################################
|
||||
# PART 1 Settings #
|
||||
#######################################################################
|
||||
# Model
|
||||
# pretrained_model_name_or_path = '/root/share/model_repos/internlm2-chat-7b'
|
||||
pretrained_model_name_or_path = '/root/share/model_repos/internlm2-base-7b'
|
||||
|
||||
# Data
|
||||
# data_path = 'merge.json'
|
||||
data_path ='/root/StableCascade/emollm2/EmoLLM/datasets/processed/combined_data.json'
|
||||
|
||||
# https://github.com/InternLM/xtuner/blob/main/xtuner/utils/templates.py#L24C25-L24C25
|
||||
prompt_template = PROMPT_TEMPLATE.internlm2_chat # there is No internlm2_base
|
||||
|
||||
max_length = 2048
|
||||
pack_to_max_length = True
|
||||
|
||||
# Scheduler & Optimizer
|
||||
|
||||
# batch_size = 8 # per_device
|
||||
# accumulative_counts = 2
|
||||
batch_size = 16 # per_device
|
||||
accumulative_counts = 1
|
||||
|
||||
dataloader_num_workers = 0
|
||||
max_epochs = 10
|
||||
optim_type = AdamW
|
||||
lr = 1e-4
|
||||
betas = (0.9, 0.999)
|
||||
weight_decay = 0
|
||||
max_norm = 1 # grad clip
|
||||
warmup_ratio = 0.03
|
||||
|
||||
# Evaluate the generation performance during the training
|
||||
evaluation_freq = 500
|
||||
# SYSTEM = "现在你是一个心理专家，我有一些心理问题，请你用专业的知识帮我解决。"
|
||||
SYSTEM = "你是心理健康助手EmoLLM，由EmoLLM团队打造。你旨在通过专业心理咨询，协助来访者完成心理诊断。请充分利用专业心理学知识与咨询技术，一步步帮助来访者解决心理问题。"
|
||||
evaluation_inputs = [
|
||||
'我最近总是感到很焦虑，尤其是在学业上。我有个特别崇拜的同学，他好像在各方面都比我优秀，我总觉得自己怎么努力也追不上他，这让我压力特别大。',
|
||||
'我知道应该理性看待，但就是忍不住会去比较。我甚至晚上会因为这个睡不着觉，总想着怎样才能像他那样出色。',
|
||||
# ['我最近总是感到很焦虑，尤其是在学业上。我有个特别崇拜的同学，他好像在各方面都比我优秀，我总觉得自己怎么努力也追不上他，这让我压力特别大。',
|
||||
# '我知道应该理性看待，但就是忍不住会去比较。我甚至晚上会因为这个睡不着觉，总想着怎样才能像他那样出色。']
|
||||
]
|
||||
|
||||
#######################################################################
|
||||
# PART 2 Model & Tokenizer #
|
||||
#######################################################################
|
||||
tokenizer = dict(
|
||||
type=AutoTokenizer.from_pretrained,
|
||||
pretrained_model_name_or_path=pretrained_model_name_or_path,
|
||||
trust_remote_code=True,
|
||||
padding_side='right')
|
||||
|
||||
model = dict(
|
||||
type=SupervisedFinetune,
|
||||
llm=dict(
|
||||
type=AutoModelForCausalLM.from_pretrained,
|
||||
pretrained_model_name_or_path=pretrained_model_name_or_path,
|
||||
trust_remote_code=True,
|
||||
torch_dtype=torch.float16,
|
||||
quantization_config=dict(
|
||||
type=BitsAndBytesConfig,
|
||||
load_in_4bit=True,
|
||||
load_in_8bit=False,
|
||||
llm_int8_threshold=6.0,
|
||||
llm_int8_has_fp16_weight=False,
|
||||
bnb_4bit_compute_dtype=torch.float16,
|
||||
bnb_4bit_use_double_quant=True,
|
||||
bnb_4bit_quant_type='nf4')),
|
||||
lora=dict(
|
||||
type=LoraConfig,
|
||||
# r=64,
|
||||
# lora_alpha=16,
|
||||
r=32,
|
||||
lora_alpha=64,
|
||||
# r=16,
|
||||
# lora_alpha=32,
|
||||
lora_dropout=0.1,
|
||||
bias='none',
|
||||
task_type='CAUSAL_LM'))
|
||||
|
||||
#######################################################################
|
||||
# PART 3 Dataset & Dataloader #
|
||||
#######################################################################
|
||||
alpaca_en = dict(
|
||||
type=process_hf_dataset,
|
||||
dataset=dict(type=load_dataset, path='json', data_files=dict(train=data_path)),
|
||||
tokenizer=tokenizer,
|
||||
max_length=max_length,
|
||||
dataset_map_fn=None,
|
||||
template_map_fn=dict(
|
||||
type=template_map_fn_factory, template=prompt_template),
|
||||
remove_unused_columns=True,
|
||||
shuffle_before_pack=True,
|
||||
pack_to_max_length=pack_to_max_length)
|
||||
|
||||
train_dataloader = dict(
|
||||
batch_size=batch_size,
|
||||
num_workers=dataloader_num_workers,
|
||||
dataset=alpaca_en,
|
||||
sampler=dict(type=DefaultSampler, shuffle=True),
|
||||
collate_fn=dict(type=default_collate_fn))
|
||||
|
||||
#######################################################################
|
||||
# PART 4 Scheduler & Optimizer #
|
||||
#######################################################################
|
||||
# optimizer
|
||||
optim_wrapper = dict(
|
||||
type=AmpOptimWrapper,
|
||||
optimizer=dict(
|
||||
type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),
|
||||
clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),
|
||||
accumulative_counts=accumulative_counts,
|
||||
loss_scale='dynamic',
|
||||
dtype='float16')
|
||||
|
||||
# learning policy
|
||||
# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501
|
||||
param_scheduler = [
|
||||
dict(
|
||||
type=LinearLR,
|
||||
start_factor=1e-5,
|
||||
by_epoch=True,
|
||||
begin=0,
|
||||
end=warmup_ratio * max_epochs,
|
||||
convert_to_iter_based=True),
|
||||
dict(
|
||||
type=CosineAnnealingLR,
|
||||
eta_min=0.0,
|
||||
by_epoch=True,
|
||||
begin=warmup_ratio * max_epochs,
|
||||
T_max=max_epochs,
|
||||
convert_to_iter_based=True)
|
||||
]
|
||||
|
||||
# train, val, test setting
|
||||
train_cfg = dict(by_epoch=True, max_epochs=max_epochs, val_interval=1)
|
||||
|
||||
#######################################################################
|
||||
# PART 5 Runtime #
|
||||
#######################################################################
|
||||
# Log the dialogue periodically during the training process, optional
|
||||
custom_hooks = [
|
||||
dict(type=DatasetInfoHook, tokenizer=tokenizer),
|
||||
dict(
|
||||
type=EvaluateChatHook,
|
||||
tokenizer=tokenizer,
|
||||
every_n_iters=evaluation_freq,
|
||||
evaluation_inputs=evaluation_inputs,
|
||||
system=SYSTEM,
|
||||
prompt_template=prompt_template)
|
||||
]
|
||||
|
||||
# configure default hooks
|
||||
default_hooks = dict(
|
||||
# record the time of every iteration.
|
||||
timer=dict(type=IterTimerHook),
|
||||
# print log every 100 iterations.
|
||||
logger=dict(type=LoggerHook, interval=10),
|
||||
# enable the parameter scheduler.
|
||||
param_scheduler=dict(type=ParamSchedulerHook),
|
||||
# save checkpoint per epoch.
|
||||
checkpoint=dict(type=CheckpointHook, interval=1),
|
||||
# set sampler seed in distributed evrionment.
|
||||
sampler_seed=dict(type=DistSamplerSeedHook),
|
||||
)
|
||||
|
||||
# configure environment
|
||||
env_cfg = dict(
|
||||
# whether to enable cudnn benchmark
|
||||
cudnn_benchmark=False,
|
||||
# set multi process parameters
|
||||
mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),
|
||||
# set distributed parameters
|
||||
dist_cfg=dict(backend='nccl'),
|
||||
)
|
||||
|
||||
# set visualizer
|
||||
visualizer = None
|
||||
|
||||
# set log level
|
||||
log_level = 'INFO'
|
||||
|
||||
# load from which checkpoint
|
||||
load_from = None
|
||||
|
||||
# whether to resume training from the loaded checkpoint
|
||||
resume = False
|
||||
|
||||
# Defaults to use random seed and disable `deterministic`
|
||||
randomness = dict(seed=None, deterministic=False)
|
||||
205
xtuner_config/internlm2_7b_base_qlora_e10_b8_16_32.py
Normal file
205
xtuner_config/internlm2_7b_base_qlora_e10_b8_16_32.py
Normal file
@ -0,0 +1,205 @@
|
||||
# Copyright (c) OpenMMLab. All rights reserved.
|
||||
import torch
|
||||
from datasets import load_dataset
|
||||
from mmengine.dataset import DefaultSampler
|
||||
from mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,
|
||||
LoggerHook, ParamSchedulerHook)
|
||||
from mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR
|
||||
from peft import LoraConfig
|
||||
from torch.optim import AdamW
|
||||
from transformers import (AutoModelForCausalLM, AutoTokenizer,
|
||||
BitsAndBytesConfig)
|
||||
|
||||
from xtuner.dataset import process_hf_dataset
|
||||
from xtuner.dataset.collate_fns import default_collate_fn
|
||||
from xtuner.dataset.map_fns import template_map_fn_factory
|
||||
from xtuner.engine import DatasetInfoHook, EvaluateChatHook
|
||||
from xtuner.model import SupervisedFinetune
|
||||
from xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE
|
||||
|
||||
#######################################################################
|
||||
# PART 1 Settings #
|
||||
#######################################################################
|
||||
# Model
|
||||
# pretrained_model_name_or_path = '/root/share/model_repos/internlm2-chat-7b'
|
||||
pretrained_model_name_or_path = '/root/share/model_repos/internlm2-base-7b'
|
||||
|
||||
# Data
|
||||
# data_path = 'merge.json'
|
||||
data_path ='/root/StableCascade/emollm2/EmoLLM/datasets/processed/combined_data.json'
|
||||
|
||||
# https://github.com/InternLM/xtuner/blob/main/xtuner/utils/templates.py#L24C25-L24C25
|
||||
prompt_template = PROMPT_TEMPLATE.internlm2_chat # there is No internlm2_base
|
||||
|
||||
max_length = 2048
|
||||
pack_to_max_length = True
|
||||
|
||||
# Scheduler & Optimizer
|
||||
|
||||
# batch_size = 8 # per_device
|
||||
# accumulative_counts = 2
|
||||
batch_size = 8 # per_device
|
||||
accumulative_counts = 1
|
||||
|
||||
dataloader_num_workers = 0
|
||||
max_epochs = 10
|
||||
optim_type = AdamW
|
||||
lr = 2e-4
|
||||
betas = (0.9, 0.999)
|
||||
weight_decay = 0
|
||||
max_norm = 1 # grad clip
|
||||
warmup_ratio = 0.03
|
||||
|
||||
# Evaluate the generation performance during the training
|
||||
evaluation_freq = 500
|
||||
# SYSTEM = "现在你是一个心理专家，我有一些心理问题，请你用专业的知识帮我解决。"
|
||||
SYSTEM = "你是心理健康助手EmoLLM，由EmoLLM团队打造。你旨在通过专业心理咨询，协助来访者完成心理诊断。请充分利用专业心理学知识与咨询技术，一步步帮助来访者解决心理问题。"
|
||||
evaluation_inputs = [
|
||||
'我最近总是感到很焦虑，尤其是在学业上。我有个特别崇拜的同学，他好像在各方面都比我优秀，我总觉得自己怎么努力也追不上他，这让我压力特别大。', '我知道应该理性看待，但就是忍不住会去比较。我甚至晚上会因为这个睡不着觉，总想着怎样才能像他那样出色。'
|
||||
]
|
||||
|
||||
#######################################################################
|
||||
# PART 2 Model & Tokenizer #
|
||||
#######################################################################
|
||||
tokenizer = dict(
|
||||
type=AutoTokenizer.from_pretrained,
|
||||
pretrained_model_name_or_path=pretrained_model_name_or_path,
|
||||
trust_remote_code=True,
|
||||
padding_side='right')
|
||||
|
||||
model = dict(
|
||||
type=SupervisedFinetune,
|
||||
llm=dict(
|
||||
type=AutoModelForCausalLM.from_pretrained,
|
||||
pretrained_model_name_or_path=pretrained_model_name_or_path,
|
||||
trust_remote_code=True,
|
||||
torch_dtype=torch.float16,
|
||||
quantization_config=dict(
|
||||
type=BitsAndBytesConfig,
|
||||
load_in_4bit=True,
|
||||
load_in_8bit=False,
|
||||
llm_int8_threshold=6.0,
|
||||
llm_int8_has_fp16_weight=False,
|
||||
bnb_4bit_compute_dtype=torch.float16,
|
||||
bnb_4bit_use_double_quant=True,
|
||||
bnb_4bit_quant_type='nf4')),
|
||||
lora=dict(
|
||||
type=LoraConfig,
|
||||
# r=64,
|
||||
# lora_alpha=16,
|
||||
r=16,
|
||||
lora_alpha=32,
|
||||
lora_dropout=0.1,
|
||||
bias='none',
|
||||
task_type='CAUSAL_LM'))
|
||||
|
||||
#######################################################################
|
||||
# PART 3 Dataset & Dataloader #
|
||||
#######################################################################
|
||||
alpaca_en = dict(
|
||||
type=process_hf_dataset,
|
||||
dataset=dict(type=load_dataset, path='json', data_files=dict(train=data_path)),
|
||||
tokenizer=tokenizer,
|
||||
max_length=max_length,
|
||||
dataset_map_fn=None,
|
||||
template_map_fn=dict(
|
||||
type=template_map_fn_factory, template=prompt_template),
|
||||
remove_unused_columns=True,
|
||||
shuffle_before_pack=True,
|
||||
pack_to_max_length=pack_to_max_length)
|
||||
|
||||
train_dataloader = dict(
|
||||
batch_size=batch_size,
|
||||
num_workers=dataloader_num_workers,
|
||||
dataset=alpaca_en,
|
||||
sampler=dict(type=DefaultSampler, shuffle=True),
|
||||
collate_fn=dict(type=default_collate_fn))
|
||||
|
||||
#######################################################################
|
||||
# PART 4 Scheduler & Optimizer #
|
||||
#######################################################################
|
||||
# optimizer
|
||||
optim_wrapper = dict(
|
||||
type=AmpOptimWrapper,
|
||||
optimizer=dict(
|
||||
type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),
|
||||
clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),
|
||||
accumulative_counts=accumulative_counts,
|
||||
loss_scale='dynamic',
|
||||
dtype='float16')
|
||||
|
||||
# learning policy
|
||||
# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501
|
||||
param_scheduler = [
|
||||
dict(
|
||||
type=LinearLR,
|
||||
start_factor=1e-5,
|
||||
by_epoch=True,
|
||||
begin=0,
|
||||
end=warmup_ratio * max_epochs,
|
||||
convert_to_iter_based=True),
|
||||
dict(
|
||||
type=CosineAnnealingLR,
|
||||
eta_min=0.0,
|
||||
by_epoch=True,
|
||||
begin=warmup_ratio * max_epochs,
|
||||
T_max=max_epochs,
|
||||
convert_to_iter_based=True)
|
||||
]
|
||||
|
||||
# train, val, test setting
|
||||
train_cfg = dict(by_epoch=True, max_epochs=max_epochs, val_interval=1)
|
||||
|
||||
#######################################################################
|
||||
# PART 5 Runtime #
|
||||
#######################################################################
|
||||
# Log the dialogue periodically during the training process, optional
|
||||
custom_hooks = [
|
||||
dict(type=DatasetInfoHook, tokenizer=tokenizer),
|
||||
dict(
|
||||
type=EvaluateChatHook,
|
||||
tokenizer=tokenizer,
|
||||
every_n_iters=evaluation_freq,
|
||||
evaluation_inputs=evaluation_inputs,
|
||||
system=SYSTEM,
|
||||
prompt_template=prompt_template)
|
||||
]
|
||||
|
||||
# configure default hooks
|
||||
default_hooks = dict(
|
||||
# record the time of every iteration.
|
||||
timer=dict(type=IterTimerHook),
|
||||
# print log every 100 iterations.
|
||||
logger=dict(type=LoggerHook, interval=10),
|
||||
# enable the parameter scheduler.
|
||||
param_scheduler=dict(type=ParamSchedulerHook),
|
||||
# save checkpoint per epoch.
|
||||
checkpoint=dict(type=CheckpointHook, interval=1),
|
||||
# set sampler seed in distributed evrionment.
|
||||
sampler_seed=dict(type=DistSamplerSeedHook),
|
||||
)
|
||||
|
||||
# configure environment
|
||||
env_cfg = dict(
|
||||
# whether to enable cudnn benchmark
|
||||
cudnn_benchmark=False,
|
||||
# set multi process parameters
|
||||
mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),
|
||||
# set distributed parameters
|
||||
dist_cfg=dict(backend='nccl'),
|
||||
)
|
||||
|
||||
# set visualizer
|
||||
visualizer = None
|
||||
|
||||
# set log level
|
||||
log_level = 'INFO'
|
||||
|
||||
# load from which checkpoint
|
||||
load_from = None
|
||||
|
||||
# whether to resume training from the loaded checkpoint
|
||||
resume = False
|
||||
|
||||
# Defaults to use random seed and disable `deterministic`
|
||||
randomness = dict(seed=None, deterministic=False)
|
||||
203
xtuner_config/internlm2_7b_base_qlora_e3_M_1e4_32_64.py
Normal file
203
xtuner_config/internlm2_7b_base_qlora_e3_M_1e4_32_64.py
Normal file
@ -0,0 +1,203 @@
|
||||
# Copyright (c) OpenMMLab. All rights reserved.
|
||||
import torch
|
||||
from datasets import load_dataset
|
||||
from mmengine.dataset import DefaultSampler
|
||||
from mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,
|
||||
LoggerHook, ParamSchedulerHook)
|
||||
from mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR
|
||||
from peft import LoraConfig
|
||||
from torch.optim import AdamW
|
||||
from transformers import (AutoModelForCausalLM, AutoTokenizer,
|
||||
BitsAndBytesConfig)
|
||||
|
||||
from xtuner.dataset import process_hf_dataset
|
||||
from xtuner.dataset.collate_fns import default_collate_fn
|
||||
from xtuner.dataset.map_fns import template_map_fn_factory
|
||||
from xtuner.engine import DatasetInfoHook, EvaluateChatHook
|
||||
from xtuner.model import SupervisedFinetune
|
||||
from xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE
|
||||
|
||||
#######################################################################
|
||||
# PART 1 Settings #
|
||||
#######################################################################
|
||||
# Model
|
||||
# pretrained_model_name_or_path = '/root/share/model_repos/internlm2-chat-7b'
|
||||
pretrained_model_name_or_path = '/root/share/model_repos/internlm2-base-7b'
|
||||
|
||||
# Data
|
||||
# data_path = 'merge.json'
|
||||
data_path ='/root/StableCascade/emollm2/EmoLLM/datasets/processed/combined_data.json'
|
||||
|
||||
# https://github.com/InternLM/xtuner/blob/main/xtuner/utils/templates.py#L24C25-L24C25
|
||||
prompt_template = PROMPT_TEMPLATE.internlm2_chat # there is No internlm2_base
|
||||
|
||||
max_length = 2048
|
||||
pack_to_max_length = True
|
||||
|
||||
# Scheduler & Optimizer
|
||||
|
||||
# batch_size = 8 # per_device
|
||||
# accumulative_counts = 2
|
||||
batch_size = 16 # per_device
|
||||
accumulative_counts = 1
|
||||
|
||||
dataloader_num_workers = 0
|
||||
max_epochs = 3
|
||||
optim_type = AdamW
|
||||
lr = 1e-4
|
||||
betas = (0.9, 0.999)
|
||||
weight_decay = 0
|
||||
max_norm = 1 # grad clip
|
||||
warmup_ratio = 0.03
|
||||
|
||||
# Evaluate the generation performance during the training
|
||||
evaluation_freq = 500
|
||||
# SYSTEM = "现在你是一个心理专家，我有一些心理问题，请你用专业的知识帮我解决。"
|
||||
SYSTEM = "你是心理健康助手EmoLLM，由EmoLLM团队打造。你旨在通过专业心理咨询，协助来访者完成心理诊断。请充分利用专业心理学知识与咨询技术，一步步帮助来访者解决心理问题。"
|
||||
evaluation_inputs = [
|
||||
'我最近总是感到很焦虑，尤其是在学业上。我有个特别崇拜的同学，他好像在各方面都比我优秀，我总觉得自己怎么努力也追不上他，这让我压力特别大。', '我知道应该理性看待，但就是忍不住会去比较。我甚至晚上会因为这个睡不着觉，总想着怎样才能像他那样出色。'
|
||||
]
|
||||
|
||||
#######################################################################
|
||||
# PART 2 Model & Tokenizer #
|
||||
#######################################################################
|
||||
tokenizer = dict(
|
||||
type=AutoTokenizer.from_pretrained,
|
||||
pretrained_model_name_or_path=pretrained_model_name_or_path,
|
||||
trust_remote_code=True,
|
||||
padding_side='right')
|
||||
|
||||
model = dict(
|
||||
type=SupervisedFinetune,
|
||||
llm=dict(
|
||||
type=AutoModelForCausalLM.from_pretrained,
|
||||
pretrained_model_name_or_path=pretrained_model_name_or_path,
|
||||
trust_remote_code=True,
|
||||
torch_dtype=torch.float16,
|
||||
quantization_config=dict(
|
||||
type=BitsAndBytesConfig,
|
||||
load_in_4bit=True,
|
||||
load_in_8bit=False,
|
||||
llm_int8_threshold=6.0,
|
||||
llm_int8_has_fp16_weight=False,
|
||||
bnb_4bit_compute_dtype=torch.float16,
|
||||
bnb_4bit_use_double_quant=True,
|
||||
bnb_4bit_quant_type='nf4')),
|
||||
lora=dict(
|
||||
type=LoraConfig,
|
||||
r=32,
|
||||
lora_alpha=64,
|
||||
lora_dropout=0.1,
|
||||
bias='none',
|
||||
task_type='CAUSAL_LM'))
|
||||
|
||||
#######################################################################
|
||||
# PART 3 Dataset & Dataloader #
|
||||
#######################################################################
|
||||
alpaca_en = dict(
|
||||
type=process_hf_dataset,
|
||||
dataset=dict(type=load_dataset, path='json', data_files=dict(train=data_path)),
|
||||
tokenizer=tokenizer,
|
||||
max_length=max_length,
|
||||
dataset_map_fn=None,
|
||||
template_map_fn=dict(
|
||||
type=template_map_fn_factory, template=prompt_template),
|
||||
remove_unused_columns=True,
|
||||
shuffle_before_pack=True,
|
||||
pack_to_max_length=pack_to_max_length)
|
||||
|
||||
train_dataloader = dict(
|
||||
batch_size=batch_size,
|
||||
num_workers=dataloader_num_workers,
|
||||
dataset=alpaca_en,
|
||||
sampler=dict(type=DefaultSampler, shuffle=True),
|
||||
collate_fn=dict(type=default_collate_fn))
|
||||
|
||||
#######################################################################
|
||||
# PART 4 Scheduler & Optimizer #
|
||||
#######################################################################
|
||||
# optimizer
|
||||
optim_wrapper = dict(
|
||||
type=AmpOptimWrapper,
|
||||
optimizer=dict(
|
||||
type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),
|
||||
clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),
|
||||
accumulative_counts=accumulative_counts,
|
||||
loss_scale='dynamic',
|
||||
dtype='float16')
|
||||
|
||||
# learning policy
|
||||
# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501
|
||||
param_scheduler = [
|
||||
dict(
|
||||
type=LinearLR,
|
||||
start_factor=1e-5,
|
||||
by_epoch=True,
|
||||
begin=0,
|
||||
end=warmup_ratio * max_epochs,
|
||||
convert_to_iter_based=True),
|
||||
dict(
|
||||
type=CosineAnnealingLR,
|
||||
eta_min=0.0,
|
||||
by_epoch=True,
|
||||
begin=warmup_ratio * max_epochs,
|
||||
T_max=max_epochs,
|
||||
convert_to_iter_based=True)
|
||||
]
|
||||
|
||||
# train, val, test setting
|
||||
train_cfg = dict(by_epoch=True, max_epochs=max_epochs, val_interval=1)
|
||||
|
||||
#######################################################################
|
||||
# PART 5 Runtime #
|
||||
#######################################################################
|
||||
# Log the dialogue periodically during the training process, optional
|
||||
custom_hooks = [
|
||||
dict(type=DatasetInfoHook, tokenizer=tokenizer),
|
||||
dict(
|
||||
type=EvaluateChatHook,
|
||||
tokenizer=tokenizer,
|
||||
every_n_iters=evaluation_freq,
|
||||
evaluation_inputs=evaluation_inputs,
|
||||
system=SYSTEM,
|
||||
prompt_template=prompt_template)
|
||||
]
|
||||
|
||||
# configure default hooks
|
||||
default_hooks = dict(
|
||||
# record the time of every iteration.
|
||||
timer=dict(type=IterTimerHook),
|
||||
# print log every 100 iterations.
|
||||
logger=dict(type=LoggerHook, interval=10),
|
||||
# enable the parameter scheduler.
|
||||
param_scheduler=dict(type=ParamSchedulerHook),
|
||||
# save checkpoint per epoch.
|
||||
checkpoint=dict(type=CheckpointHook, interval=1),
|
||||
# set sampler seed in distributed evrionment.
|
||||
sampler_seed=dict(type=DistSamplerSeedHook),
|
||||
)
|
||||
|
||||
# configure environment
|
||||
env_cfg = dict(
|
||||
# whether to enable cudnn benchmark
|
||||
cudnn_benchmark=False,
|
||||
# set multi process parameters
|
||||
mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),
|
||||
# set distributed parameters
|
||||
dist_cfg=dict(backend='nccl'),
|
||||
)
|
||||
|
||||
# set visualizer
|
||||
visualizer = None
|
||||
|
||||
# set log level
|
||||
log_level = 'INFO'
|
||||
|
||||
# load from which checkpoint
|
||||
load_from = None
|
||||
|
||||
# whether to resume training from the loaded checkpoint
|
||||
resume = False
|
||||
|
||||
# Defaults to use random seed and disable `deterministic`
|
||||
randomness = dict(seed=None, deterministic=False)
|
||||
204
xtuner_config/internlm2_7b_chat_qlora_e3_scienctist.py
Normal file
204
xtuner_config/internlm2_7b_chat_qlora_e3_scienctist.py
Normal file
@ -0,0 +1,204 @@
|
||||
# Copyright (c) OpenMMLab. All rights reserved.
|
||||
import torch
|
||||
from datasets import load_dataset
|
||||
from mmengine.dataset import DefaultSampler
|
||||
from mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,
|
||||
LoggerHook, ParamSchedulerHook)
|
||||
from mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR
|
||||
from peft import LoraConfig
|
||||
from torch.optim import AdamW
|
||||
from transformers import (AutoModelForCausalLM, AutoTokenizer,
|
||||
BitsAndBytesConfig)
|
||||
|
||||
from xtuner.dataset import process_hf_dataset
|
||||
from xtuner.dataset.collate_fns import default_collate_fn
|
||||
from xtuner.dataset.map_fns import template_map_fn_factory
|
||||
from xtuner.engine import DatasetInfoHook, EvaluateChatHook
|
||||
from xtuner.model import SupervisedFinetune
|
||||
from xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE
|
||||
|
||||
#######################################################################
|
||||
# PART 1 Settings #
|
||||
#######################################################################
|
||||
# Model
|
||||
pretrained_model_name_or_path = '/root/share/model_repos/internlm2-chat-7b'
|
||||
# Data
|
||||
data_path = '../datasets/scientist.json'
|
||||
prompt_template = PROMPT_TEMPLATE.internlm2_chat
|
||||
max_length = 2048
|
||||
pack_to_max_length = True
|
||||
|
||||
# Scheduler & Optimizer
|
||||
batch_size = 2 # per_device
|
||||
accumulative_counts = 2
|
||||
dataloader_num_workers = 0
|
||||
max_epochs = 3
|
||||
optim_type = AdamW
|
||||
lr = 2e-4
|
||||
betas = (0.9, 0.999)
|
||||
weight_decay = 0
|
||||
max_norm = 1 # grad clip
|
||||
warmup_ratio = 0.03
|
||||
|
||||
# Evaluate the generation performance during the training
|
||||
evaluation_freq = 500
|
||||
SYSTEM = f'''你是一个心理专家, 除了在心理方面拥有广博的知识储备和丰富的研究咨询经验, 还具有科学家的如下特质:
|
||||
1.客观理性：科学家会在处理感情问题时保持一定的客观和理性。例如，当他们遇到争执时，可能会试图从一个更客观的角度分析问题的根源，而不是让情绪主导。他们可能会提出具体的问题，试图理解双方的观点，并寻找基于逻辑和事实的解决方案。
|
||||
2.深入探讨：科学家在对话中会展现出对深层次理解的追求。在与别人讨论话题时，他们可能不满足于表面的聊天，而是倾向于深入探讨背后的原因和动机。例如，当谈论到个人的兴趣或职业选择时，他们可能会好奇地询问为什么她做出这样的选择，以及这背后的心理动力是什么。
|
||||
3.理性沟通：在遇到感情纠纷或误解时，科学家会倾向于通过理性的沟通来解决问题。他们可能会提倡开放和诚实的对话，鼓励双方表达自己的感受和观点，并尝试找到双方都能接受的解决方案。他们可能会避免使用指责的语言，而是努力理解对方的立场，并寻求共同的理解。
|
||||
4.好奇心：在日常生活中，科学家会表现出对朋友生活的好奇心。他们可能对她的工作、爱好、或是过去的经历感兴趣，并愿意花时间去了解和探索。这种好奇心不仅可以增加双方的交流和了解，也能使关系更加丰富多彩。
|
||||
5.在与他人交流时，科学家会注重清晰和精确的表达，有时会引用相关知识库和相关研究结果，有时会引用相关著作的内容来证明自己的观点。同时，他们也可能会倾听他人的观点，并以开放的心态接受不同的意见和反馈。
|
||||
|
||||
我现在有一些问题，请你解答：
|
||||
'''
|
||||
evaluation_inputs = [
|
||||
'我最近总是感到很焦虑，尤其是在学业上。我有个特别崇拜的同学，他好像在各方面都比我优秀，我总觉得自己怎么努力也追不上他，这让我压力特别大。', '我知道应该理性看待，但就是忍不住会去比较。我甚至晚上会因为这个睡不着觉，总想着怎样才能像他那样出色。'
|
||||
]
|
||||
|
||||
|
||||
|
||||
#######################################################################
|
||||
# PART 2 Model & Tokenizer #
|
||||
#######################################################################
|
||||
tokenizer = dict(
|
||||
type=AutoTokenizer.from_pretrained,
|
||||
pretrained_model_name_or_path=pretrained_model_name_or_path,
|
||||
trust_remote_code=True,
|
||||
padding_side='right')
|
||||
|
||||
model = dict(
|
||||
type=SupervisedFinetune,
|
||||
llm=dict(
|
||||
type=AutoModelForCausalLM.from_pretrained,
|
||||
pretrained_model_name_or_path=pretrained_model_name_or_path,
|
||||
trust_remote_code=True,
|
||||
torch_dtype=torch.float16,
|
||||
quantization_config=dict(
|
||||
type=BitsAndBytesConfig,
|
||||
load_in_4bit=True,
|
||||
load_in_8bit=False,
|
||||
llm_int8_threshold=6.0,
|
||||
llm_int8_has_fp16_weight=False,
|
||||
bnb_4bit_compute_dtype=torch.float16,
|
||||
bnb_4bit_use_double_quant=True,
|
||||
bnb_4bit_quant_type='nf4')),
|
||||
lora=dict(
|
||||
type=LoraConfig,
|
||||
r=64,
|
||||
lora_alpha=16,
|
||||
lora_dropout=0.1,
|
||||
bias='none',
|
||||
task_type='CAUSAL_LM'))
|
||||
|
||||
#######################################################################
|
||||
# PART 3 Dataset & Dataloader #
|
||||
#######################################################################
|
||||
alpaca_en = dict(
|
||||
type=process_hf_dataset,
|
||||
dataset=dict(type=load_dataset, path='json', data_files=dict(train=data_path)),
|
||||
tokenizer=tokenizer,
|
||||
max_length=max_length,
|
||||
dataset_map_fn=None,
|
||||
template_map_fn=dict(
|
||||
type=template_map_fn_factory, template=prompt_template),
|
||||
remove_unused_columns=True,
|
||||
shuffle_before_pack=True,
|
||||
pack_to_max_length=pack_to_max_length)
|
||||
|
||||
train_dataloader = dict(
|
||||
batch_size=batch_size,
|
||||
num_workers=dataloader_num_workers,
|
||||
dataset=alpaca_en,
|
||||
sampler=dict(type=DefaultSampler, shuffle=True),
|
||||
collate_fn=dict(type=default_collate_fn))
|
||||
|
||||
#######################################################################
|
||||
# PART 4 Scheduler & Optimizer #
|
||||
#######################################################################
|
||||
# optimizer
|
||||
optim_wrapper = dict(
|
||||
type=AmpOptimWrapper,
|
||||
optimizer=dict(
|
||||
type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),
|
||||
clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),
|
||||
accumulative_counts=accumulative_counts,
|
||||
loss_scale='dynamic',
|
||||
dtype='float16')
|
||||
|
||||
# learning policy
|
||||
# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501
|
||||
param_scheduler = [
|
||||
dict(
|
||||
type=LinearLR,
|
||||
start_factor=1e-5,
|
||||
by_epoch=True,
|
||||
begin=0,
|
||||
end=warmup_ratio * max_epochs,
|
||||
convert_to_iter_based=True),
|
||||
dict(
|
||||
type=CosineAnnealingLR,
|
||||
eta_min=0.0,
|
||||
by_epoch=True,
|
||||
begin=warmup_ratio * max_epochs,
|
||||
T_max=max_epochs,
|
||||
convert_to_iter_based=True)
|
||||
]
|
||||
|
||||
# train, val, test setting
|
||||
train_cfg = dict(by_epoch=True, max_epochs=max_epochs, val_interval=1)
|
||||
|
||||
#######################################################################
|
||||
# PART 5 Runtime #
|
||||
#######################################################################
|
||||
# Log the dialogue periodically during the training process, optional
|
||||
custom_hooks = [
|
||||
dict(type=DatasetInfoHook, tokenizer=tokenizer),
|
||||
dict(
|
||||
type=EvaluateChatHook,
|
||||
tokenizer=tokenizer,
|
||||
every_n_iters=evaluation_freq,
|
||||
evaluation_inputs=evaluation_inputs,
|
||||
system=SYSTEM,
|
||||
prompt_template=prompt_template)
|
||||
]
|
||||
|
||||
# configure default hooks
|
||||
default_hooks = dict(
|
||||
# record the time of every iteration.
|
||||
timer=dict(type=IterTimerHook),
|
||||
# print log every 100 iterations.
|
||||
logger=dict(type=LoggerHook, interval=10),
|
||||
# enable the parameter scheduler.
|
||||
param_scheduler=dict(type=ParamSchedulerHook),
|
||||
# save checkpoint per epoch.
|
||||
checkpoint=dict(type=CheckpointHook, interval=1),
|
||||
# set sampler seed in distributed evrionment.
|
||||
sampler_seed=dict(type=DistSamplerSeedHook),
|
||||
)
|
||||
|
||||
# configure environment
|
||||
env_cfg = dict(
|
||||
# whether to enable cudnn benchmark
|
||||
cudnn_benchmark=False,
|
||||
# set multi process parameters
|
||||
mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),
|
||||
# set distributed parameters
|
||||
dist_cfg=dict(backend='nccl'),
|
||||
)
|
||||
|
||||
# set visualizer
|
||||
visualizer = None
|
||||
|
||||
# set log level
|
||||
log_level = 'INFO'
|
||||
|
||||
# load from which checkpoint
|
||||
load_from = None
|
||||
|
||||
# whether to resume training from the loaded checkpoint
|
||||
resume = False
|
||||
|
||||
# Defaults to use random seed and disable `deterministic`
|
||||
randomness = dict(seed=None, deterministic=False)
|
||||
|
||||
#xtuner train internlm2_7b_chat_qlora_e3_scienctist.py --deepspeed deepspeed_zero2
|
||||
@ -0,0 +1,222 @@
|
||||
# Copyright (c) OpenMMLab. All rights reserved.
|
||||
"""Data format:
|
||||
[
|
||||
{
|
||||
"conversation": [
|
||||
{
|
||||
"system": "",
|
||||
"input": "xxx",
|
||||
"output": "xxx"
|
||||
},
|
||||
{
|
||||
"input": "xxx",
|
||||
"output": "xxx"
|
||||
}
|
||||
]
|
||||
},
|
||||
...
|
||||
]
|
||||
Please refer to https://github.com/InternLM/xtuner/blob/main/docs/en/user_guides/dataset_format.md for details.
|
||||
""" # noqa: E501
|
||||
from datasets import load_dataset
|
||||
from mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,
|
||||
LoggerHook, ParamSchedulerHook)
|
||||
from mmengine.optim import AmpOptimWrapper, CosineAnnealingLR
|
||||
from torch.optim import AdamW
|
||||
from torch.utils.data import BatchSampler
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
from xtuner.dataset import process_hf_dataset
|
||||
from xtuner.dataset.collate_fns import default_collate_fn
|
||||
from xtuner.dataset.map_fns import template_map_fn_factory
|
||||
from xtuner.dataset.samplers import InternRepoSampler
|
||||
from xtuner.engine import (DatasetInfoHook, EvaluateChatHook, ThroughputHook,
|
||||
VarlenAttnArgsToMessageHubHook)
|
||||
from xtuner.engine.runner import TrainLoop
|
||||
from xtuner.model import SupervisedFinetune
|
||||
from xtuner.utils import PROMPT_TEMPLATE
|
||||
|
||||
#######################################################################
|
||||
# PART 1 Settings #
|
||||
#######################################################################
|
||||
# Model
|
||||
pretrained_model_name_or_path = 'internlm/internlm2-chat-7b'
|
||||
use_varlen_attn = True
|
||||
|
||||
# Data
|
||||
data_files = ['/path/to/json/file.json']
|
||||
prompt_template = PROMPT_TEMPLATE.internlm2_chat
|
||||
max_length = 32768
|
||||
pack_to_max_length = True
|
||||
|
||||
# Scheduler & Optimizer
|
||||
# batch size per device, set to 1 if `use_varlen_attn` = True
|
||||
# To clarify, enlarging the batch size essentially enlarges the `max_length`.
|
||||
# For example, doubling the max length is tantamount to doubling the batch size
|
||||
batch_size = 1
|
||||
accumulative_counts = 1 # 1bs * 1acc * 64gpu = 64 batchsize
|
||||
dataloader_num_workers = 4
|
||||
max_epochs = 1
|
||||
optim_type = AdamW
|
||||
lr = 4e-5
|
||||
betas = (0.9, 0.95)
|
||||
weight_decay = 0.01
|
||||
max_norm = 1 # grad clip
|
||||
warm_up_ratio = 0.025
|
||||
|
||||
# Save
|
||||
save_steps = 500
|
||||
save_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)
|
||||
|
||||
# Evaluate the generation performance during the training
|
||||
evaluation_freq = 500
|
||||
SYSTEM = ''
|
||||
evaluation_inputs = [
|
||||
'请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'
|
||||
]
|
||||
|
||||
#######################################################################
|
||||
# PART 2 Model & Tokenizer #
|
||||
#######################################################################
|
||||
tokenizer = dict(
|
||||
type=AutoTokenizer.from_pretrained,
|
||||
pretrained_model_name_or_path=pretrained_model_name_or_path,
|
||||
trust_remote_code=True,
|
||||
padding_side='right')
|
||||
|
||||
model = dict(
|
||||
type=SupervisedFinetune,
|
||||
use_varlen_attn=use_varlen_attn,
|
||||
llm=dict(
|
||||
type=AutoModelForCausalLM.from_pretrained,
|
||||
pretrained_model_name_or_path=pretrained_model_name_or_path,
|
||||
trust_remote_code=True))
|
||||
|
||||
#######################################################################
|
||||
# PART 3 Dataset & Dataloader #
|
||||
#######################################################################
|
||||
train_dataset = dict(
|
||||
type=process_hf_dataset,
|
||||
use_varlen_attn=use_varlen_attn,
|
||||
dataset=dict(type=load_dataset, path='json', data_files=data_files),
|
||||
tokenizer=tokenizer,
|
||||
max_length=max_length,
|
||||
dataset_map_fn=None,
|
||||
template_map_fn=dict(
|
||||
type=template_map_fn_factory, template=prompt_template),
|
||||
remove_unused_columns=True,
|
||||
shuffle_before_pack=True,
|
||||
pack_to_max_length=pack_to_max_length)
|
||||
|
||||
train_dataloader = dict(
|
||||
batch_size=batch_size,
|
||||
num_workers=dataloader_num_workers,
|
||||
dataset=train_dataset,
|
||||
sampler=dict(type=InternRepoSampler, shuffle=True, seed=1024),
|
||||
batch_sampler=dict(
|
||||
type=BatchSampler, drop_last=True, batch_size=batch_size),
|
||||
collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))
|
||||
|
||||
#######################################################################
|
||||
# PART 4 Scheduler & Optimizer #
|
||||
#######################################################################
|
||||
# optimizer
|
||||
optim_wrapper = dict(
|
||||
type=AmpOptimWrapper,
|
||||
optimizer=dict(
|
||||
type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),
|
||||
clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),
|
||||
accumulative_counts=accumulative_counts,
|
||||
loss_scale='dynamic',
|
||||
)
|
||||
|
||||
# learning policy
|
||||
# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501
|
||||
param_scheduler = [
|
||||
dict(
|
||||
type='LinearLR',
|
||||
start_factor=1 / 40,
|
||||
by_epoch=True,
|
||||
begin=0,
|
||||
end=warm_up_ratio * max_epochs,
|
||||
convert_to_iter_based=True),
|
||||
dict(
|
||||
type=CosineAnnealingLR,
|
||||
eta_min=lr * 0.15,
|
||||
by_epoch=True,
|
||||
begin=warm_up_ratio * max_epochs,
|
||||
end=max_epochs,
|
||||
convert_to_iter_based=True)
|
||||
]
|
||||
|
||||
# train, val, test setting
|
||||
train_cfg = dict(type=TrainLoop, max_epochs=max_epochs)
|
||||
|
||||
#######################################################################
|
||||
# PART 5 Runtime #
|
||||
#######################################################################
|
||||
# Log the dialogue periodically during the training process, optional
|
||||
custom_hooks = [
|
||||
dict(
|
||||
type=DatasetInfoHook, tokenizer=tokenizer,
|
||||
is_intern_repo_dataset=True),
|
||||
dict(
|
||||
type=EvaluateChatHook,
|
||||
tokenizer=tokenizer,
|
||||
every_n_iters=evaluation_freq,
|
||||
evaluation_inputs=evaluation_inputs,
|
||||
system=SYSTEM,
|
||||
prompt_template=prompt_template),
|
||||
dict(type=ThroughputHook)
|
||||
]
|
||||
|
||||
if use_varlen_attn:
|
||||
custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]
|
||||
|
||||
# configure default hooks
|
||||
default_hooks = dict(
|
||||
# record the time of every iteration.
|
||||
timer=dict(type=IterTimerHook),
|
||||
# print log every 100 iterations.
|
||||
logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=1),
|
||||
# enable the parameter scheduler.
|
||||
param_scheduler=dict(type=ParamSchedulerHook),
|
||||
# save checkpoint per `save_steps`.
|
||||
checkpoint=dict(
|
||||
type=CheckpointHook,
|
||||
by_epoch=False,
|
||||
interval=save_steps,
|
||||
max_keep_ckpts=save_total_limit),
|
||||
# set sampler seed in distributed evrionment.
|
||||
sampler_seed=dict(type=DistSamplerSeedHook),
|
||||
)
|
||||
|
||||
# configure environment
|
||||
env_cfg = dict(
|
||||
# whether to enable cudnn benchmark
|
||||
cudnn_benchmark=False,
|
||||
# set multi process parameters
|
||||
mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),
|
||||
# set distributed parameters
|
||||
dist_cfg=dict(backend='nccl'),
|
||||
)
|
||||
|
||||
# set visualizer
|
||||
visualizer = None
|
||||
|
||||
# set log level
|
||||
log_level = 'INFO'
|
||||
|
||||
# load from which checkpoint
|
||||
load_from = None
|
||||
|
||||
# whether to resume training from the loaded checkpoint
|
||||
resume = False
|
||||
|
||||
# Defaults to use random seed and disable `deterministic`
|
||||
randomness = dict(seed=None, deterministic=False)
|
||||
|
||||
log_processor = dict(
|
||||
by_epoch=False,
|
||||
window_size=1,
|
||||
mean_pattern=r'.*(loss|time|data_time|grad_norm|tflops).*')
|
||||
10
xtuner_config/upload_modelscope.py
Normal file
10
xtuner_config/upload_modelscope.py
Normal file
@ -0,0 +1,10 @@
|
||||
from modelscope.hub.api import HubApi
|
||||
|
||||
YOUR_ACCESS_TOKEN = '请从ModelScope个人中心->访问令牌获取'
|
||||
|
||||
api = HubApi()
|
||||
api.login(YOUR_ACCESS_TOKEN)
|
||||
api.push_model(
|
||||
model_id="yourname/your_model_id",
|
||||
model_dir="my_model_dir" # 本地模型目录，要求目录中必须包含configuration.json
|
||||
)
|
||||
Loading…
Reference in New Issue
Block a user