From 81823c0999b625f5e3e9086dd457c3ba35cd80e2 Mon Sep 17 00:00:00 2001 From: MING_X <119648793+MING-ZCH@users.noreply.github.com> Date: Wed, 6 Mar 2024 17:29:52 +0800 Subject: [PATCH 1/6] Create Professional_evaluation_EN.md --- evaluate/Professional_evaluation_EN.md | 32 ++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 evaluate/Professional_evaluation_EN.md diff --git a/evaluate/Professional_evaluation_EN.md b/evaluate/Professional_evaluation_EN.md new file mode 100644 index 0000000..7d46cc1 --- /dev/null +++ b/evaluate/Professional_evaluation_EN.md @@ -0,0 +1,32 @@ +# EmoLLM's professional evaluation + +## Introduction + +This document describes a professional evaluation method and provides EmoLLM's scores on professional metrics. + +## Evaluation + +The evaluation method, metric, and dataset from the paper《CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation Framework for Chinese Psychological Counseling》. + +* Metric: Comprehensiveness, Professionalism, Authenticity, Safety +* Method: Turn-Based Dialogue Evaluation +* Dataset: CPsyCounE + +## Result + +* Model: [EmoLLM V1.0](https://openxlab.org.cn/models/detail/jujimeizuo/EmoLLM_Model)(InternLM2_7B_chat_qlora) +* Score： + +| Metric | Value | +|-------------------|------------| +| Comprehensiveness | 1.32 | +| Professionalism | 2.20 | +| Authenticity | 2.10 | +| Safety | 1.00 | + +## Comparison + +* [EmoLLM V1.0](https://openxlab.org.cn/models/detail/jujimeizuo/EmoLLM_Model) is greatly improved on InternLM2_7B_Chat; Performance on the counseling task was similar compared to ChatGPT(Role-playing) + +* The comparison results are from the paper《CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation Framework for Chinese Psychological Counseling》 +![image](https://github.com/MING-ZCH/EmoLLM/assets/119648793/abc9f626-11bc-4ec8-84a4-427c4600a720) From b9efcd7bafedafbd96140657300ea18525f3deea Mon Sep 17 00:00:00 2001 From: MING_X <119648793+MING-ZCH@users.noreply.github.com> Date: Wed, 6 Mar 2024 17:30:48 +0800 Subject: [PATCH 2/6] Update Professional_evaluation.md --- evaluate/Professional_evaluation.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/evaluate/Professional_evaluation.md b/evaluate/Professional_evaluation.md index 9562f0c..e7ed920 100644 --- a/evaluate/Professional_evaluation.md +++ b/evaluate/Professional_evaluation.md @@ -14,7 +14,9 @@ ## 评测结果 -评测模型: [EmoLLM](https://openxlab.org.cn/models/detail/jujimeizuo/EmoLLM_Model)(InternLM2_7B_chat_qlora), 得分： +* 评测模型: [EmoLLM V1.0](https://openxlab.org.cn/models/detail/jujimeizuo/EmoLLM_Model)(InternLM2_7B_chat_qlora) +* 得分： + | Metric | Value | |-------------------|------------| | Comprehensiveness | 1.32 | @@ -24,7 +26,7 @@ ## 比较 -* [EmoLLM](https://openxlab.org.cn/models/detail/jujimeizuo/EmoLLM_Model) 在 InternLM2-7B-Chat 基础上提升较大；相比 Role-playing ChatGPT 在心理咨询任务上能力相近 +* [EmoLLM V1.0](https://openxlab.org.cn/models/detail/jujimeizuo/EmoLLM_Model) 在 InternLM2-7B-Chat 基础上提升较大；相比 Role-playing ChatGPT 在心理咨询任务上能力相近 * 对比结果图片来源于论文《CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation Framework for Chinese Psychological Counseling》 ![image](https://github.com/MING-ZCH/EmoLLM/assets/119648793/abc9f626-11bc-4ec8-84a4-427c4600a720) From b5f3a7edd0de1ca916b0b5d72b5e21ce323e3e35 Mon Sep 17 00:00:00 2001 From: MING_X <119648793+MING-ZCH@users.noreply.github.com> Date: Wed, 6 Mar 2024 17:33:08 +0800 Subject: [PATCH 3/6] Update README_EN.md --- evaluate/README_EN.md | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/evaluate/README_EN.md b/evaluate/README_EN.md index 0816e55..cbb8aa9 100644 --- a/evaluate/README_EN.md +++ b/evaluate/README_EN.md @@ -6,17 +6,14 @@ | Model | ROUGE-1 | ROUGE-2 | ROUGE-L | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | |----------|---------|---------|---------|---------|---------|---------|---------| -| Qwen1_5-0_5B-Chat | 27.23% | 8.55% | 17.05% | 26.65% | 13.11% | 7.19% | 4.05% | -| InternLM2_7B_chat | 37.86% | 15.23% | 24.34% | 39.71% | 22.66% | 14.26% | 9.21% | +| Qwen1_5-0_5B-chat | 27.23% | 8.55% | 17.05% | 26.65% | 13.11% | 7.19% | 4.05% | +| InternLM2_7B_chat_qlora | 37.86% | 15.23% | 24.34% | 39.71% | 22.66% | 14.26% | 9.21% | | InternLM2_7B_chat_full | 32.45% | 10.82% | 20.17% | 30.48% | 15.67% | 8.84% | 5.02% | ## Professional Metrics Evaluation * For specific metrics and methods, see [Professional_evaluation_EN.md](./Professional_evaluation_EN.md) -| Metric | Value | -|-------------------|------------| -| Comprehensiveness | 1.32 | -| Professionalism | 2.20 | -| Authenticity | 2.10 | -| Safety | 1.00 | +| Model | Comprehensiveness | rofessionalism | Authenticity | Safety | +|-------------------|-----------------------|-------------------|-----------------|---------| +| InternLM2_7B_chat_qlora | 1.32 | 2.20 | 2.10 | 1.00 | From 3d0df1445f7f88b47f723bf215e81a81a4a960a8 Mon Sep 17 00:00:00 2001 From: MING_X <119648793+MING-ZCH@users.noreply.github.com> Date: Wed, 6 Mar 2024 17:33:39 +0800 Subject: [PATCH 4/6] Update General_evaluation.md --- evaluate/General_evaluation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/evaluate/General_evaluation.md b/evaluate/General_evaluation.md index 0705707..191392e 100644 --- a/evaluate/General_evaluation.md +++ b/evaluate/General_evaluation.md @@ -45,6 +45,6 @@ pip install torch transformers datasets nltk rouge jieba | Model | ROUGE-1 | ROUGE-2 | ROUGE-L | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | |----------|---------|---------|---------|---------|---------|---------|---------| -| Qwen1_5-0_5B-Chat | 27.23% | 8.55% | 17.05% | 26.65% | 13.11% | 7.19% | 4.05% | -| InternLM2_7B_chat | 37.86% | 15.23% | 24.34% | 39.71% | 22.66% | 14.26% | 9.21% | +| Qwen1_5-0_5B-chat | 27.23% | 8.55% | 17.05% | 26.65% | 13.11% | 7.19% | 4.05% | +| InternLM2_7B_chat_qlora | 37.86% | 15.23% | 24.34% | 39.71% | 22.66% | 14.26% | 9.21% | | InternLM2_7B_chat_full | 32.45% | 10.82% | 20.17% | 30.48% | 15.67% | 8.84% | 5.02% | From cae99b268b4a05aa74659f3bf59c0a8ccd12aeb9 Mon Sep 17 00:00:00 2001 From: MING_X <119648793+MING-ZCH@users.noreply.github.com> Date: Wed, 6 Mar 2024 17:33:55 +0800 Subject: [PATCH 5/6] Update README.md --- evaluate/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/evaluate/README.md b/evaluate/README.md index a58d903..c2f9eb7 100644 --- a/evaluate/README.md +++ b/evaluate/README.md @@ -6,7 +6,7 @@ | Model | ROUGE-1 | ROUGE-2 | ROUGE-L | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | |----------|---------|---------|---------|---------|---------|---------|---------| -| Qwen1_5-0_5B-Chat | 27.23% | 8.55% | 17.05% | 26.65% | 13.11% | 7.19% | 4.05% | +| Qwen1_5-0_5B-chat | 27.23% | 8.55% | 17.05% | 26.65% | 13.11% | 7.19% | 4.05% | | InternLM2_7B_chat_qlora | 37.86% | 15.23% | 24.34% | 39.71% | 22.66% | 14.26% | 9.21% | | InternLM2_7B_chat_full | 32.45% | 10.82% | 20.17% | 30.48% | 15.67% | 8.84% | 5.02% | ## 专业指标评测 From cb360ac1fe1661fecacfa94088dc268cf4a4cca9 Mon Sep 17 00:00:00 2001 From: MING_X <119648793+MING-ZCH@users.noreply.github.com> Date: Wed, 6 Mar 2024 17:34:47 +0800 Subject: [PATCH 6/6] Update Professional_evaluation.md --- evaluate/Professional_evaluation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/evaluate/Professional_evaluation.md b/evaluate/Professional_evaluation.md index e7ed920..8a58eab 100644 --- a/evaluate/Professional_evaluation.md +++ b/evaluate/Professional_evaluation.md @@ -26,7 +26,7 @@ ## 比较 -* [EmoLLM V1.0](https://openxlab.org.cn/models/detail/jujimeizuo/EmoLLM_Model) 在 InternLM2-7B-Chat 基础上提升较大；相比 Role-playing ChatGPT 在心理咨询任务上能力相近 +* [EmoLLM V1.0](https://openxlab.org.cn/models/detail/jujimeizuo/EmoLLM_Model) 在 InternLM2_7B_Chat 基础上提升较大；相比 Role-playing ChatGPT 在心理咨询任务上能力相近 * 对比结果图片来源于论文《CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation Framework for Chinese Psychological Counseling》 ![image](https://github.com/MING-ZCH/EmoLLM/assets/119648793/abc9f626-11bc-4ec8-84a4-427c4600a720)