bakaEC/OliveSensorAPI
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request #44 from ZeyuBa/evaluate

Browse Source 
add InternLM2_7B_chat_full eval
This commit is contained in:
এ許我辞忧࿐♡ 2024-03-03 23:18:56 +08:00 committed by GitHub
commit 0128bd42c2
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
4 changed files with 9 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
evaluate/Genera_evaluation.md
View File

@ -47,3 +47,4 @@ pip install torch transformers datasets nltk rouge jieba
|----------|---------|---------|---------|---------|---------|---------|---------|
| Qwen1_5-0_5B-Chat | 27.23% | 8.55% | 17.05% | 26.65% | 13.11% | 7.19% | 4.05% |
| InternLM2_7B_chat | 37.86% | 15.23% | 24.34% | 39.71% | 22.66% | 14.26% | 9.21% |
| InternLM2_7B_chat_full | 32.45% | 10.82% | 20.17% | 30.48% | 15.67% | 8.84% | 5.02% |

2
evaluate/InternLM2_7B_chat_eval.py
View File

@ -102,7 +102,7 @@ for batch in dataloader:
verbose=False,
errors='replace'
).replace("医生：","") for i in range(batch_size)]
hypotheses.extend([r.replace(stop_word," ").split()[0] for r in batch_response if stop_word in r])
hypotheses.extend([r.replace(stop_word," ").split()[0] if stop_word in r else r for r in batch_response])
# Load metric

4
evaluate/README.md
View File

@ -7,8 +7,8 @@
| Model | ROUGE-1 | ROUGE-2 | ROUGE-L | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 |
|----------|---------|---------|---------|---------|---------|---------|---------|
| Qwen1_5-0_5B-Chat | 27.23% | 8.55% | 17.05% | 26.65% | 13.11% | 7.19% | 4.05% |
| InternLM2_7B_chat | 37.86% | 15.23% | 24.34% | 39.71% | 22.66% | 14.26% | 9.21% |
| InternLM2_7B_chat_qlora | 37.86% | 15.23% | 24.34% | 39.71% | 22.66% | 14.26% | 9.21% |
| InternLM2_7B_chat_full | 32.45% | 10.82% | 20.17% | 30.48% | 15.67% | 8.84% | 5.02% |
## 专业指标评测
* 具体指标、方法见 [Professional_evaluation.md](./Professional_evaluation.md)

14
evaluate/README_EN.md
View File

@ -4,15 +4,11 @@
* For specific metrics and methods, see [General_evaluation.md](./General_evaluation_EN.md)
| Metric | Value |
|---------|----------------------|
| ROUGE-1 | 27.23% |
| ROUGE-2 | 8.55% |
| ROUGE-L | 17.05% |
| BLEU-1 | 26.65% |
| BLEU-2 | 13.11% |
| BLEU-3 | 7.19% |
| BLEU-4 | 4.05% |
| Model | ROUGE-1 | ROUGE-2 | ROUGE-L | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 |
|----------|---------|---------|---------|---------|---------|---------|---------|
| Qwen1_5-0_5B-Chat | 27.23% | 8.55% | 17.05% | 26.65% | 13.11% | 7.19% | 4.05% |
| InternLM2_7B_chat | 37.86% | 15.23% | 24.34% | 39.71% | 22.66% | 14.26% | 9.21% |
| InternLM2_7B_chat_full | 32.45% | 10.82% | 20.17% | 30.48% | 15.67% | 8.84% | 5.02% |
## Professional Metrics Evaluation