diff --git a/README.md b/README.md index 5416a9b..be413e0 100644 --- a/README.md +++ b/README.md @@ -78,7 +78,9 @@ ### 🎇最近更新 -- 【2024.3.12】在百度飞浆平台发布[艾薇](https://aistudio.baidu.com/community/app/63335) +- 【2024.3.25】在百度飞桨平台发布[爹系男友心理咨询师](https://aistudio.baidu.com/community/app/68787) +- 【2024.3.24】在OpenXLab和ModelScope平台发布InternLM2-Base-7B QLoRA微调模型, 具体请查看[InternLM2-Base-7B QLoRA](./xtuner_config/README_internlm2_7b_base_qlora.md) +- 【2024.3.12】在百度飞桨平台发布[艾薇](https://aistudio.baidu.com/community/app/63335) - 【2024.3.11】 **EmoLLM V2.0 相比 EmoLLM V1.0 全面提升,已超越 Role-playing ChatGPT 在心理咨询任务上的能力!**[点击体验EmoLLM V2.0](https://openxlab.org.cn/apps/detail/Farewell1/EmoLLMV2.0),更新[数据集统计及详细信息](./datasets/)、[路线图](./assets/Roadmap_ZH.png) - 【2024.3.9】 新增并发功能加速 [QA 对生成](./scripts/qa_generation/)、[RAG pipeline](./rag/) - 【2024.3.3】 [基于InternLM2-7B-chat全量微调版本EmoLLM V2.0开源](https://openxlab.org.cn/models/detail/ajupyter/EmoLLM_internlm2_7b_full),需要两块A100*80G,更新专业评估,详见[evaluate](./evaluate/),更新基于PaddleOCR的PDF转txt工具脚本,详见[scripts](./scripts/) @@ -236,7 +238,7 @@ git clone https://github.com/SmartFlowAI/EmoLLM.git | [ZeyuBa](https://github.com/ZeyuBa) | 自动化所在读硕士 | | | | [aiyinyuedejustin](https://github.com/aiyinyuedejustin) | 宾夕法尼亚大学在读硕士 | | | | [Nobody-ML](https://github.com/Nobody-ML) | 中国石油大学(华东)在读本科生 | | | -| [chg0901](https://github.com/chg0901) | [MiniSora](https://github.com/mini-sora/minisora/) |[MiniSora](https://github.com/mini-sora/minisora/)主要维护者,管理员| LLM微调、数据清洗、文档翻译 | +| [chg0901](https://github.com/chg0901) | [MiniSora](https://github.com/mini-sora/minisora/) |[MiniSora](https://github.com/mini-sora/minisora/)主要维护者,管理员| LLM预训练和微调、模型上传、数据清洗、文档翻译 | | [Mxoder](https://github.com/Mxoder) | 北京航空航天大学在读本科生 | | | | [Anooyman](https://github.com/Anooyman) | 南京理工大学硕士 | | | | [Vicky-3021](https://github.com/Vicky-3021) | 西安电子科技大学硕士(研0) | | | diff --git a/README_EN.md b/README_EN.md index eaa0b03..28aad03 100644 --- a/README_EN.md +++ b/README_EN.md @@ -77,8 +77,12 @@ The Model aims to fully understand and promote the mental health of individuals, - Psychological resilience: Refers to an individual's ability to recover from adversity and adapt. Those with strong psychological resilience can bounce back from challenges and learn and grow from them. - Prevention and intervention measures: The Mental Health Grand Model also includes strategies for preventing psychological issues and promoting mental health, such as psychological education, counseling, therapy, and social support systems. - Assessment and diagnostic tools: Effective promotion of mental health requires scientific tools to assess individuals' psychological states and diagnose potential psychological issues. + ### Recent Updates -- 【2024.3.12】 Released on Baidu Flying Pulp Platform [aiwei](https://aistudio.baidu.com/community/app/63335) + +- 【2024.3.25】 [Daddy-like Boy-Friend] is released on Baidu Paddle-Paddle AI Studio Platform (https://aistudio.baidu.com/community/app/68787) +- 【2024.3.24】 The InternLM2-Base-7B QLoRA fine-tuned model has been released on the OpenXLab and ModelScope platforms. For more details, please refer to [InternLM2-Base-7B QLoRA](./xtuner_config/README_internlm2_7b_base_qlora.md). +- 【2024.3.12】 [aiwei] is released on Baidu Paddle-Paddle AI Studio Platform (https://aistudio.baidu.com/community/app/63335) - 【2024.3.11】 **EmoLLM V2.0 is greatly improved in all scores compared to EmoLLM V1.0. Surpasses the performance of Role-playing ChatGPT on counseling tasks!** [Click to experience EmoLLM V2.0](https://openxlab.org.cn/apps/detail/Farewell1/EmoLLMV2.0), update [dataset statistics and details](./datasets/), [Roadmap](./assets/Roadmap_ZH.png) - 【2024.3.9】 Add concurrency acceleration [QA pair generation](./scripts/qa_generation/), [RAG pipeline](./rag/) - 【2024.3.3】 [Based on InternLM2-7B-chat full fine-tuned version EmoLLM V2.0 open sourced](https://openxlab.org.cn/models/detail/ajupyter/EmoLLM_internlm2_7b_full), need two A100*80G, update professional evaluation, see [evaluate](./evaluate/), update PaddleOCR-based PDF to txt tool scripts, see [scripts](./scripts/). @@ -248,7 +252,7 @@ This project uses Git for version control. You can see the currently available v | [ZeyuBa](https://github.com/ZeyuBa) | Institute of Automation, Master's student | | | | [aiyinyuedejustin](https://github.com/aiyinyuedejustin) | University of Pennsylvania, Master's student | | | | [Nobody-ML](https://github.com/Nobody-ML) | China University of Petroleum (East China), Undergraduate student | | | -| [chg0901](https://github.com/chg0901) | [MiniSora](https://github.com/mini-sora/minisora) |Maintainer and Admin of [MiniSora](https://github.com/mini-sora/minisora) | LLM Fine-Tuning, Data Cleaning and Docs Translation | +| [chg0901](https://github.com/chg0901) | [MiniSora](https://github.com/mini-sora/minisora) |Maintainer and Admin of [MiniSora](https://github.com/mini-sora/minisora) | LLM Pre-Training and Fine-Tuning, Model Uploading, Data Cleaning and Docs Translation | | [Mxoder](https://github.com/Mxoder) | Beihang University, Undergraduate student | | | | [Anooyman](https://github.com/Anooyman) | Nanjing University of Science and Technology, Master's student | | | | [Vicky-3021](https://github.com/Vicky-3021) | Xidian University, Master's student (Research Year 0) | | | diff --git a/datasets/README.md b/datasets/README.md index a2c2385..bd0fcf9 100644 --- a/datasets/README.md +++ b/datasets/README.md @@ -4,11 +4,13 @@ * 数据按格式分为两种类型:**QA** 和 **Conversation** * 数据汇总:General(**6个数据集**);Role-play(**3个数据集**) - ## 数据集类型 +## 数据集类型 + * **General**:通用数据集,包含心理学知识、心理咨询技术等通用内容 * **Role-play**:角色扮演数据集,包含特定角色对话风格数据等内容 ## 数据类型 + * **QA**:问答对 * **Conversation**:多轮对话 @@ -28,7 +30,9 @@ | …… | …… | …… | …… | ## 数据集来源 -**General**: + +### **General** + * 数据集 data 来自本项目 * 数据集 data_pro 来自本项目 * 数据集 multi_turn_dataset_1 来源 [Smile](https://github.com/qiuhuachuan/smile) @@ -36,24 +40,28 @@ * 数据集 single_turn_dataset_1 来自本项目 * 数据集 single_turn_dataset_2 来自本项目 -**Role-play**: +### **Role-play** + * 数据集 aiwei 来自本项目 * 数据集 tiangou 来自本项目 * 数据集 SoulStar 来源 [SoulStar](https://github.com/Nobody-ML/SoulStar) ## 数据集去重 + 结合绝对匹配以及模糊匹配(Simhash)算法,对数据集进行去重以提升微调模型的效果。在确保数据集的高质量的同时,通过调整阈值减少因错误匹配而丢失重要数据的风险。 -**Simhash算法介绍** +### **Simhash算法介绍** + Simhash(相似性哈希)是一种用于检测大量数据中相似或重复项的算法。它通过将文本转换为一组数值指纹来工作,这些指纹对相似的文本具有高度的相似性。Simhash算法对于处理文本数据特别有效,尤其是在处理大量数据时。 -**Simhash实现步骤** +### **Simhash实现步骤** + *文本预处理:将文本数据转换为适合Simhash处理的格式。这可能包括分词、去除停用词、词干提取等。 *生成Simhash指纹:对预处理后的文本应用Simhash算法,生成一组数值指纹。每个指纹代表文本内容的一个哈希值。 *比较指纹:通过比较哈希值的相似性来识别重复或相似的记录。Simhash的特点是即使在文本有少量差异时,生成的哈希值也具有较高的相似性。 *确定阈值:设置一个相似性阈值,只有当两个指纹的相似度超过这个阈值时,才认为它们代表相似或重复的记录。 *处理相似记录:对于被标记为相似的记录,可以进一步人工审查或自动合并,以消除重复。 -## 用法 -### deduplicate.py -`deduplicate.py` 用于将datasets下以模型命名的文件夹下(例如:'datasets/qwen').json数据进行去重,输出去重后的数据到 `datasets/qwen/dedup` 文件夹下。 \ No newline at end of file +### deduplicate.py用法 + +`deduplicate.py` 用于将datasets下以模型命名的文件夹下(例如:'datasets/qwen').json数据进行去重,输出去重后的数据到 `datasets/qwen/dedup` 文件夹下。 diff --git a/scripts/README_Model_Uploading.md b/scripts/README_Model_Uploading.md new file mode 100644 index 0000000..682a785 --- /dev/null +++ b/scripts/README_Model_Uploading.md @@ -0,0 +1,235 @@ +# 模型上传指南 + +## OpenXLab浦源平台 + +### OpenXLab平台介绍 + +