年翻更新

- 全新ui
- 全面优化websocket逻辑，提高数字人和ui连接的稳定性及资源开销
- 全面优化唤醒逻辑，提供稳定的普通唤醒模式和前置词唤醒模式
- 优化拾音质量，支持多声道麦克风拾音
- 优化自动播放服务器的对接机制，提供稳定和兼容旧版ue工程的对接模式
- 数字人接口输出机器人表情，以适应新fay ui及单片机的数字人表情输出
- 使用更高级的音频时长计算方式，可以更精准控制音频播放完成后的逻辑
- 修复点击关闭按钮会导致程序退出的bug
- 修复没有麦克风的设备开启麦克风会出错的问题
- 为服务器主机地址提供配置项，以方便服务器部署

.gitignore

@@ -0,0 +1,129 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/

README.md

@@ -1,100 +1,119 @@
+[`English`](https://github.com/xszyou/Fay/blob/main/README_EN.md) 
+
 <div align="center">
     <br>
-    <img src="images/icon.png" alt="Fay" />
-    <h1>Fay开源数字人框架</h1></div>
+    <img src="readme/icon.png" alt="Fay">
+    <h1>FAY</h1>
+	<h3>Fay数字人框架</h3>
+</div>
+
+！！重要通知：我们会在2024年12月31日前把Fay的三个版本合并成1个，并致力提供更稳定更全面的功能。
+
+我们致力于思考面向终端的数字人落地应用，并通过完整代码把思考结果呈现给大家。Fay数字人框架，向上适配各种数字人模型技术，向下接入各式大语言模型，并且便于更换诸如TTS、ASR等模型，为单片机、app、网站提供全面的数字人应用接口。      
+
+
+
+## **功能特点**
+
+
+
+- 完全开源，商用免责
+- 支持全离线使用
+- 支持毫秒级回复
+- 自由匹配数字人模型、大语言模型、ASR、TTS模型
+- 支持数字人自动播报模式（虚拟教师、虚拟主播、新闻播报）
+- 支持任意终端使用：单片机、app、网站、大屏、成熟系统接入等
+- 支持多用户多路并发
+- 提供文字沟通接口、声音沟通接口、数字人模型接口、管理控制接口、自动播放接口
+- 支持语音指令灵活配置执行
+- 支持自定义知识库、自定义问答对、自定义人设信息
+- 支持唤醒及打断对话
+- 支持服务器及单机模式
+- 支持机器人表情输出
+
+###               
+
+## **Fay数字人框架**
+
+![](readme/chat.png)
+
+![](readme/controller.png)
 
 
 
 
 
-                   
 
-如果你需要的是一个人机交互的数字人助理（当然，你也可以命令它开关设备）或者需要把数字人集成到你的产品上，请移步 [`助理完整版`](https://github.com/TheRamU/Fay/tree/fay-assistant-edition)
+## **源码启动**
 
-如果你需要是一个可以自主决策、主动联系主人的agent，请移步[`agent版`](https://github.com/TheRamU/Fay/tree/fay-agent-edition)
 
-如果你需要是一个线上线下的销售员，请移步[`带货完整版`](https://github.com/TheRamU/Fay/tree/fay-sales-edition)    
+### **环境** 
+- Python 3.9、3.10、3.11、3.12
+- Windows、macos、linux
 
-使用文档：https://qqk9ntwbcit.feishu.cn/wiki/space/7321626901586411523
+### **安装依赖**
+
+```shell
+pip install -r requirements.txt
+```
+
+### **配置**
++ 依照说明修改 `./system.conf` 文件
+
+### **启动**
+启动Fay控制器
+```shell
+python main.py
+```
 
 
 
-“用数字人去改变成熟传统软件的交互逻辑”
+## **或docker 启动**
+
+1. 下载助理版
+https://github.com/xszyou/Fay
+2.  修改 `./system.conf` 文件
+3. 删除requirements.txt下pyqt5~=5.15.6
+
+build （修改配置文件后，需要重新build）
+```shell
+docker build  -t fay ./fay-assistant-edition
+```
+run
+```shell
+docker run -it --rm -p 5000:5000 -p 10001:10001 -p 10002:10002 -p 10003:10003 fay
+```
 
 
-Fay数字人2024.10.16更新：
+## **后续**
 
-Fay-助理版
-
-1、更正默认配置；
-
-2、去除麦克风选项（使用系统麦克风）；
-
-3、去掉 eyes(为全新在cv架构做准备)；
-
-4、新增文字语音穿透功能（为统一带货版和agent版做好准备。穿透功能就是把输入的文字直接输出到面板、声音及声音沟通接口和数字人接口，把输入语音直接输出到面板、声音及声音沟通接口和数字人接口。）
-
-4、ollama提示词对接；
-
-5、修复Azure无法修改音色问题；
-
-6、修复wsa_server websocket客户端管理的逻辑问题；
-
-7、修复修复wsa_server websocket线程同步问题。
-
-Fay数字人2024.10.09更新：
-
-🌟Fay-助理版
-
-1、 优化文字沟通接口的流式输出逻辑
-
--- fay的文字沟通接口，按标点符号切割并通过http stream返回，这样做语音合成时，能够完整处理每个断句的语音情绪。
-
-2、 去掉内置ngrok.cc内网穿透代码
-
--- ngrok内网穿透可以让普通pc当作服务器使用，让移动端或者智能设备随时与fay通讯。如需继续使用可以外部启动ngrok或者其他穿透客户端，效果是一样的。
-
-3、优化ASR处理速度
-
--- VAD（语音活动检测）时间由700ms减小到200ms，可以降低fay识别到我们已经说完一句话的时间，从而让fay更快作出响应
-
-4、优化TTS速度
-
--- azure不使用ssml明显加速，使用azure tts平均时间可以减小700ms以上
-
--- 修复本地播放完声音再发送音频给数字人的bug，可以让面板播放音频更快让数字人作出响应（虽然不太可能本地播放和数字人播放同时使用）
-
--- 语音合成之前替换掉“*”，这是大语言模型经常作出的返回，非常影响语音合成的用户体验
-
-5、优化Q&A文件的应用逻辑
-
--- 文件格式由excel更换成csv，可以更好兼容linux环境
-
--- 配置上Q&A文件之后会自动缓存大语言模型回复，相同对话的回复时间可以降到1ms以下
-
--- csv的第3列可以配置执行脚本，可以实现RPA操作或对智能硬件的控制
-
-6、完善是否做语音合成的逻辑
-
--- 只有在需要发送远程音频或者发送给数字人或者面板播放时才合成音频，避免资源的浪费
-
-7、修正多用户同时与fay聊天时qa日志有可能混乱的问题
-
-8、 修复fay_core.py上的变量（usernmae）错识导致的远程音频传输出错
-
-9、修复pygame init时无扬声器导致出错
-
-10、去掉面板出现了"完成!"、“远程音频设备连接上”、“远程音频输入输出设备已经断开”、“服务已关闭！”等不必要的日志信息
-
-🌟Fay-UE5：
-
-- 5.4工程，与fay的对接方式更新为流式对接
-
---会从fay小段文字接收然后做tts处理，这样可以更快速作出响应。
+![interface](readme\interface.png)
 
 
 
-更多更新日志：https://qqk9ntwbcit.feishu.cn/wiki/UlbZwfAXgiKSquk52AkcibhHngg
+### ***使用数字人（非必须）***
 
-联系我们，请关注微信公众号 Fay数字人 
+ue: https://github.com/xszyou/fay-ue5
 
+unity：https://qqk9ntwbcit.feishu.cn/wiki/Se9xw04hUiss00kb2Lmci1BVnM9
+
+metahuman-stream（2d）：https://qqk9ntwbcit.feishu.cn/wiki/Ik1kwO9X5iilnGkFwRhcnmtvn3e
+
+duix（android)：https://qqk9ntwbcit.feishu.cn/wiki/Ik1kwO9X5iilnGkFwRhcnmtvn3e()
+
+aibote(windows cpu克隆人)：[‬‍‬‍⁠‍‍‍‌https://qqk9ntwbcit.feishu.cn/wiki/ULaywzVRti0HXWkhCzacoSPAnIg
+
+
+
+### ***集成到自家产品（非必须）***
+接口：https://qqk9ntwbcit.feishu.cn/wiki/Mcw3wbA3RiNZzwkexz6cnKCsnhh
+
+
+
+### **联系**
+
+**商务QQ: 467665317**
+
+**交流群及资料教程**关注公众号 **fay数字人**（**请先star本仓库**）
+
+![](readme/gzh.jpg)

ai_module/baidu_emotion.py

@@ -0,0 +1,99 @@
+import json
+import requests
+import time
+from core.authorize_tb import Authorize_Tb
+from utils import config_util as cfg
+from utils import util
+
+def get_sentiment(cont):
+    emotion = Emotion()
+    answer = emotion.get_sentiment(cont)
+    return answer
+
+class Emotion:
+
+    def __init__(self):
+        self.app_id = cfg.baidu_emotion_app_id
+        self.authorize_tb = Authorize_Tb()
+
+    def get_sentiment(self, cont):
+        token = self.__check_token()
+        if token is None or token == 'expired':
+            token_info = self.__get_token()
+            if token_info is not None and  token_info['access_token']  is not None:
+                #转换过期时间
+                updated_in_seconds = time.time()
+                expires_timedelta = token_info['expires_in']
+                expiry_timestamp_in_seconds = updated_in_seconds + expires_timedelta
+                expiry_timestamp_in_milliseconds = expiry_timestamp_in_seconds * 1000
+                if token == 'expired':
+                    self.authorize_tb.update_by_userid(self.app_id, token_info['access_token'], expiry_timestamp_in_milliseconds)
+                else:
+                    self.authorize_tb.add(self.app_id, token_info['access_token'], expiry_timestamp_in_milliseconds)
+                token = token_info['access_token']
+            else:
+                token = None
+   
+        if token is not None:
+            try:
+                url=f"https://aip.baidubce.com/rpc/2.0/nlp/v1/sentiment_classify?access_token={token}"
+                req = json.dumps({"text": cont})
+                headers = {
+                        'Content-Type': 'application/json',
+                        'Accept': 'application/json'
+                    }                
+                r = requests.post(url, headers=headers,  data=req)
+                if r.status_code != 200:
+                    util.log(1, f"百度情感分析对接有误: {r.text}")
+                    return 0
+                info = json.loads(r.text)
+                if not self.has_field(info,'error_code'):
+                    return info['items'][0]['sentiment']
+                else:
+                    util.log(1, f"百度情感分析对接有误： {info['error_msg']}") 
+                    return 0
+            except Exception as e:
+                util.log(1, f"百度情感分析对接有误： {str(e)}")           
+                return 0
+        else:
+            return 0
+
+    def __check_token(self):
+        self.authorize_tb.init_tb()
+        info = self.authorize_tb.find_by_userid(self.app_id)
+        if info is not None:
+            if info[1] >= int(time.time())*1000:
+                return info[0]
+            else:
+                return 'expired'
+        else:
+            return None 
+
+    def __get_token(self):
+        try:
+            url=f"https://aip.baidubce.com/oauth/2.0/token?client_id={cfg.baidu_emotion_api_key}&client_secret={cfg.baidu_emotion_secret_key}&grant_type=client_credentials"            
+            headers = {'Content-Type':'application/json;charset=UTF-8'}
+            r = requests.post(url, headers=headers)    
+            if r.status_code != 200:
+                info = json.loads(r.text)
+                if info["error"] == "invalid_client":
+                    util.log(1, f"请检查baidu_emotion_api_key")
+                else:
+                    util.log(1, f"请检查baidu_emotion_secret_key")
+                return None            
+            info = json.loads(r.text)
+            if not self.has_field(info,'error_code'):
+                return info
+            else:
+                util.log(1, f"百度情感分析对接有误： {info['error_msg']}") 
+                util.log(1, f"请检查baidu_emotion_api_key和baidu_emotion_secret_key") 
+                return None
+        except Exception as e:
+            util.log(1, f"百度情感分析有1误： {str(e)}")
+            return None
+
+    
+    def has_field(self, array, field):
+        return any(field in item for item in array)
+  
+

ai_module/nlp_cemotion.py

@@ -0,0 +1,10 @@
+
+def get_sentiment(c,text):
+    try:
+        return c.predict(text)
+    except BaseException as e:
+                print("请稍后")
+                print(e)
+
+
+

ai_module/yolov8.py

@@ -0,0 +1,148 @@
+from ultralytics import YOLO
+from scipy.spatial import procrustes
+import numpy as np
+import cv2
+import time
+from scheduler.thread_manager import MyThread
+
+__fei_eyes = None
+class FeiEyes:
+    
+    def __init__(self):
+    
+        """   
+        鼻子（0）
+        左眼（1），右眼（2）
+        左耳（3），右耳（4）
+        左肩（5），右肩（6）
+        左肘（7），右肘（8）
+        左腕（9），右腕（10）
+        左髋（11），右髋（12）
+        左膝（13），右膝（14）
+        左脚踝（15），右脚踝（16）
+        """
+        self.POSE_PAIRS = [ 
+        (3, 5),  (5, 6),  # upper body
+        (5, 7), (6, 8), (7, 9), (8, 10),  # lower body
+        (11, 12), (11, 13), (12, 14), (13, 15)  # arms
+        ]
+        self.my_face = np.array([[154.4565, 193.7006],
+                [181.8575, 164.8366],
+                [117.1820, 164.3602],
+                [213.5605, 193.0460],
+                [ 62.7056, 193.5217]])
+        self.is_running = False
+        self.img = None
+        
+    def is_sitting(self, keypoints):
+        if len(keypoints) < 17:  # 确保有足够的关键点
+            return False
+        # 检查每个关键点的置信度
+        if keypoints[11][2] < 0.5 or keypoints[12][2] < 0.5 or keypoints[13][2] < 0.5 or keypoints[14][2] < 0.5 or keypoints[15][2] < 0.5 or keypoints[16][2] < 0.5:
+            return False
+
+        left_hip, right_hip = keypoints[11][:2], keypoints[12][:2]
+        left_knee, right_knee = keypoints[13][:2], keypoints[14][:2]
+        left_ankle, right_ankle = keypoints[15][:2], keypoints[16][:2]
+
+        hip_knee_y = (left_hip[1] + right_hip[1] + left_knee[1] + right_knee[1]) / 4
+        knee_ankle_y = (left_knee[1] + right_knee[1] + left_ankle[1] + right_ankle[1]) / 4
+
+        return hip_knee_y < knee_ankle_y
+
+    def is_standing(self, keypoints):
+        if len(keypoints) < 17 or keypoints[0][2] < 0.5 or keypoints[15][2] < 0.5 or keypoints[16][2] < 0.5:
+            return False
+
+        head = keypoints[0][:2]
+        left_ankle, right_ankle = keypoints[15][:2], keypoints[16][:2]
+
+        return head[1] > left_ankle[1] and head[1] > right_ankle[1]
+
+    def get_counts(self):
+        if not self.is_running:
+            return 0,0,0
+        return self.person_count, self.stand_count, self.sit_count
+
+    def get_status(self):
+        return self.is_running
+    
+    def get_img(self):
+        if self.is_running:
+            return self.img
+        else:
+            return None
+    
+    def start(self):
+        cap = cv2.VideoCapture(0)
+        if cap.isOpened():
+            self.is_running = True
+            MyThread(target=self.run, args=[cap]).start()
+
+    def stop(self):
+        self.is_running = False
+
+    def run(self, cap):
+         model = YOLO("yolov8n-pose.pt")
+         while self.is_running:
+            time.sleep(0.033)
+            ret, frame = cap.read()
+            self.img = frame
+            operated_frame = frame.copy()
+            if not ret:
+                break
+            results = model.predict(operated_frame, verbose=False)
+            person_count = 0
+            sit_count = 0
+            stand_count = 0
+            for res in results:  # loop over results
+                for box, cls in zip(res.boxes.xyxy, res.boxes.cls):  # loop over detections
+                    x1, y1, x2, y2 = box
+                    cv2.rectangle(operated_frame, (int(x1.item()), int(y1.item())), (int(x2.item()), int(y2.item())), (0, 255, 0), 2)
+                    cv2.putText(operated_frame, f"{res.names[int(cls.item())]}", (int(x1.item()), int(y1.item()) - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)        
+                if res.keypoints is not None and res.keypoints.xy.numel() > 0: # check if keypoints exist
+                    keypoints = res.keypoints[0]
+                    #总人数
+                    person_count += 1
+                    #坐着的人数
+                    if self.is_sitting(keypoints):
+                        sit_count += 1
+                    #站着的人数
+                    elif self.is_standing(keypoints):
+                        stand_count += 1
+
+                    for keypoint in keypoints:  # loop over keypoints
+                       
+                        if len(keypoint) == 3: 
+                            x, y, conf = keypoint
+                            if conf > 0.5:  # draw keypoints with confidence greater than 0.5
+                                cv2.circle(operated_frame, (int(x.item()), int(y.item())), 3, (0, 0, 255), -1)
+
+                    # Draw lines connecting keypoints
+                    for pair in self.POSE_PAIRS:
+                         if pair[0] < len(keypoints) and pair[1] < len(keypoints):
+                            pt1, pt2 = keypoints[pair[0]][:2], keypoints[pair[1]][:2]
+                            conf1, conf2 = keypoints[pair[0]][2], keypoints[pair[1]][2]
+                            if conf1 > 0.5 and conf2 > 0.5:
+                            # cv2.line(operated_frame, (int(pt1[0].item()), int(pt1[1].item())), (int(pt2[0].item()), int(pt2[1].item())), (255, 255, 0), 2)
+                                pass
+            self.person_count = person_count
+            self.sit_count = sit_count
+            self.stand_count = stand_count
+            cv2.imshow("YOLO v8 Fay Eyes", operated_frame)
+            cv2.waitKey(1)
+
+         cap.release()
+         cv2.destroyAllWindows()
+
+
+def new_instance():
+    global __fei_eyes
+    if __fei_eyes is None:
+        __fei_eyes = FeiEyes()
+    return __fei_eyes
+
+
+
+        
+

asr/ali_nls.py

@@ -0,0 +1,188 @@
+from threading import Thread
+from threading import Lock
+import websocket
+import json
+import time
+import ssl
+import wave
+import _thread as thread
+from aliyunsdkcore.client import AcsClient
+from aliyunsdkcore.request import CommonRequest
+
+from core import wsa_server
+from scheduler.thread_manager import MyThread
+from utils import util
+from utils import config_util as cfg
+from core.authorize_tb import Authorize_Tb
+
+__running = True
+__my_thread = None
+_token = ''
+
+
+def __post_token():
+    global _token
+    __client = AcsClient(
+        cfg.key_ali_nls_key_id,
+        cfg.key_ali_nls_key_secret,
+        "cn-shanghai"
+    )
+
+    __request = CommonRequest()
+    __request.set_method('POST')
+    __request.set_domain('nls-meta.cn-shanghai.aliyuncs.com')
+    __request.set_version('2019-02-28')
+    __request.set_action_name('CreateToken')
+    info = json.loads(__client.do_action_with_exception(__request))
+    _token = info['Token']['Id']
+    authorize = Authorize_Tb()
+    authorize_info = authorize.find_by_userid(cfg.key_ali_nls_key_id)
+    if authorize_info is not None:
+       authorize.update_by_userid(cfg.key_ali_nls_key_id, _token, info['Token']['ExpireTime']*1000)
+    else:
+       authorize.add(cfg.key_ali_nls_key_id, _token, info['Token']['ExpireTime']*1000) 
+
+def __runnable():
+    while __running:
+        __post_token()
+        time.sleep(60 * 60 * 12)
+
+
+def start():
+    MyThread(target=__runnable).start()
+
+
+class ALiNls:
+    # 初始化
+    def __init__(self, username):
+        self.__URL = 'wss://nls-gateway-cn-shenzhen.aliyuncs.com/ws/v1'
+        self.__ws = None
+        self.__frames = []
+        self.__closing = False
+        self.__task_id = ''
+        self.done = False
+        self.finalResults = ""
+        self.username = username
+        self.data = b''
+        self.__endding = False
+        self.__is_close = False
+        self.lock = Lock()
+
+    def __create_header(self, name):
+        if name == 'StartTranscription':
+            self.__task_id = util.random_hex(32)
+        header = {
+            "appkey": cfg.key_ali_nls_app_key,
+            "message_id": util.random_hex(32),
+            "task_id": self.__task_id,
+            "namespace": "SpeechTranscriber",
+            "name": name
+        }
+        return header
+
+    # 收到websocket消息的处理
+    def on_message(self, ws, message):
+        try:
+            data = json.loads(message)
+            header = data['header']
+            name = header['name']
+            if name == 'SentenceEnd':
+                self.done = True
+                self.finalResults = data['payload']['result']
+                if wsa_server.get_web_instance().is_connected(self.username):
+                    wsa_server.get_web_instance().add_cmd({"panelMsg": self.finalResults, "Username" : self.username})
+                if wsa_server.get_instance().is_connected(self.username):
+                    content = {'Topic': 'Unreal', 'Data': {'Key': 'log', 'Value': self.finalResults}, 'Username' : self.username}
+                    wsa_server.get_instance().add_cmd(content)
+                ws.close()#TODO
+            elif name == 'TranscriptionResultChanged':
+                self.finalResults = data['payload']['result']
+                if wsa_server.get_web_instance().is_connected(self.username):
+                    wsa_server.get_web_instance().add_cmd({"panelMsg": self.finalResults, "Username" : self.username})
+                if wsa_server.get_instance().is_connected(self.username):
+                    content = {'Topic': 'Unreal', 'Data': {'Key': 'log', 'Value': self.finalResults}, 'Username' : self.username}
+                    wsa_server.get_instance().add_cmd(content)
+
+        except Exception as e:
+            print(e)
+        # print("### message:", message)
+
+    # 收到websocket的关闭要求
+    def on_close(self, ws, code, msg):
+        self.__endding = True
+        self.__is_close = True
+        if msg:
+            print("aliyun asr服务不太稳定:", msg)
+
+    # 收到websocket错误的处理
+    def on_error(self, ws, error):
+        print("aliyun asr error:", error)
+
+    # 收到websocket连接建立的处理
+    def on_open(self, ws):
+        self.__endding = False
+        #为了兼容多路asr，关闭过程数据
+        def run(*args):
+            while self.__endding == False:
+                try: 
+                    if len(self.__frames) > 0:
+                        with self.lock:
+                            frame = self.__frames.pop(0)
+                        if isinstance(frame, dict):
+                            ws.send(json.dumps(frame))
+                        elif isinstance(frame, bytes):
+                            ws.send(frame, websocket.ABNF.OPCODE_BINARY)
+                            self.data += frame
+                    else:
+                        time.sleep(0.001)  # 避免忙等
+                except Exception as e:
+                    print(e)
+                    break
+            if self.__is_close == False:
+                for frame in self.__frames:
+                    ws.send(frame, websocket.ABNF.OPCODE_BINARY)
+                frame = {"header": self.__create_header('StopTranscription')}
+                ws.send(json.dumps(frame))
+        thread.start_new_thread(run, ())
+
+    def __connect(self):
+        self.finalResults = ""
+        self.done = False
+        with self.lock:
+            self.__frames.clear()
+        self.__ws = websocket.WebSocketApp(self.__URL + '?token=' + _token, on_message=self.on_message)
+        self.__ws.on_open = self.on_open
+        self.__ws.on_error = self.on_error
+        self.__ws.on_close = self.on_close
+        self.__ws.run_forever(sslopt={"cert_reqs": ssl.CERT_NONE})
+
+    def send(self, buf):
+        with self.lock:
+            self.__frames.append(buf)
+
+    def start(self):
+        Thread(target=self.__connect, args=[]).start()
+        data = {
+            'header': self.__create_header('StartTranscription'),
+            "payload": {
+                "format": "pcm",
+                "sample_rate": 16000,
+                "enable_intermediate_result": True,
+                "enable_punctuation_prediction": False,
+                "enable_inverse_text_normalization": True,
+                "speech_noise_threshold": -1
+            }
+        }
+        self.send(data)
+
+    def end(self):
+        self.__endding = True
+        with wave.open('cache_data/input2.wav', 'wb') as wf:
+            # 设置音频参数
+            n_channels = 1  # 单声道
+            sampwidth = 2   # 16 位音频，每个采样点 2 字节
+            wf.setnchannels(n_channels)
+            wf.setsampwidth(sampwidth)
+            wf.setframerate(16000)
+            wf.writeframes(self.data)
+        self.data = b''

asr/funasr.py

@@ -0,0 +1,142 @@
+"""
+感谢北京中科大脑神经算法工程师张聪聪提供funasr集成代码
+"""
+from threading import Thread
+import websocket
+import json
+import time
+import ssl
+import _thread as thread
+
+from core import wsa_server
+from utils import config_util as cfg
+from utils import util
+
+class FunASR:
+    # 初始化
+    def __init__(self, username):
+        self.__URL = "ws://{}:{}".format(cfg.local_asr_ip, cfg.local_asr_port)
+        self.__ws = None
+        self.__connected = False
+        self.__frames = []
+        self.__state = 0
+        self.__closing = False
+        self.__task_id = ''
+        self.done = False
+        self.finalResults = ""
+        self.__reconnect_delay = 1
+        self.__reconnecting = False
+        self.username = username
+
+    
+    # 收到websocket消息的处理
+    def on_message(self, ws, message):
+        try:
+            self.done = True
+            self.finalResults = message
+            if wsa_server.get_web_instance().is_connected(self.username):
+                wsa_server.get_web_instance().add_cmd({"panelMsg": self.finalResults, "Username" : self.username})
+            if wsa_server.get_instance().is_connected(self.username):
+                content = {'Topic': 'Unreal', 'Data': {'Key': 'log', 'Value': self.finalResults}, 'Username' : self.username}
+                wsa_server.get_instance().add_cmd(content)
+   
+        except Exception as e:
+            print(e)
+
+        if self.__closing:
+            try:
+                self.__ws.close()
+            except Exception as e:
+                print(e)
+
+    # 收到websocket错误的处理
+    def on_close(self, ws, code, msg):
+        self.__connected = False
+        # util.printInfo(1, self.username, f"### CLOSE:{msg}")
+        self.__ws = None
+
+    # 收到websocket错误的处理
+    def on_error(self, ws, error):
+        self.__connected = False
+        # util.printInfo(1, self.username, f"### error:{error}")
+        self.__ws = None
+
+    #重连
+    def __attempt_reconnect(self):
+        if not self.__reconnecting:
+            self.__reconnecting = True
+            # util.log(1, "尝试重连funasr...")
+            while not self.__connected:
+                time.sleep(self.__reconnect_delay)
+                self.start()
+                self.__reconnect_delay *= 2  
+            self.__reconnect_delay = 1  
+            self.__reconnecting = False
+
+
+    # 收到websocket连接建立的处理
+    def on_open(self, ws):
+        self.__connected = True
+
+        def run(*args):
+            while self.__connected:
+                try:
+                    if len(self.__frames) > 0:
+                        frame = self.__frames[0]
+
+                        self.__frames.pop(0)
+                        if type(frame) == dict:
+                            ws.send(json.dumps(frame))
+                        elif type(frame) == bytes:
+                            ws.send(frame, websocket.ABNF.OPCODE_BINARY)
+                        # print('发送 ------> ' + str(type(frame)))
+                except Exception as e:
+                    print(e)
+                time.sleep(0.04)
+
+        thread.start_new_thread(run, ())
+
+    def __connect(self):
+        self.finalResults = ""
+        self.done = False
+        self.__frames.clear()
+        websocket.enableTrace(False)
+        self.__ws = websocket.WebSocketApp(self.__URL, on_message=self.on_message,on_close=self.on_close,on_error=self.on_error,subprotocols=["binary"])
+        self.__ws.on_open = self.on_open
+
+        self.__ws.run_forever(sslopt={"cert_reqs": ssl.CERT_NONE})
+
+    def add_frame(self, frame):
+        self.__frames.append(frame)
+
+    def send(self, buf):
+        self.__frames.append(buf)
+
+    def send_url(self, url):
+        frame = {'url' : url}
+        self.__ws.send(json.dumps(frame))
+
+    def start(self):
+        Thread(target=self.__connect, args=[]).start()
+        data = {
+                'vad_need':False,
+                'state':'StartTranscription'
+        }
+        self.add_frame(data)
+
+    def end(self):
+        if self.__connected:
+            try:
+                for frame in self.__frames:
+                    self.__frames.pop(0)
+                    if type(frame) == dict:
+                        self.__ws.send(json.dumps(frame))
+                    elif type(frame) == bytes:
+                        self.__ws.send(frame, websocket.ABNF.OPCODE_BINARY)
+                self.__frames.clear()
+                frame = {'vad_need':False,'state':'StopTranscription'}
+                self.__ws.send(json.dumps(frame))
+            except Exception as e:
+                print(e)
+        self.__closing = True
+        self.__connected = False

asr/funasr/ASR_client.py

@@ -0,0 +1,74 @@
+import pyaudio
+import websockets
+import asyncio
+from queue import Queue
+import argparse
+import json
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--host", type=str, default="127.0.0.1", required=False, help="host ip, localhost, 0.0.0.0")
+parser.add_argument("--port", type=int, default=10197, required=False, help="grpc server port")
+parser.add_argument("--chunk_size", type=int, default=160, help="ms")
+parser.add_argument("--vad_needed", type=bool, default=True)
+args = parser.parse_args()
+
+voices = Queue()
+
+async def record():
+    global voices 
+    FORMAT = pyaudio.paInt16
+    CHANNELS = 1
+    RATE = 16000
+    CHUNK = int(RATE / 1000 * args.chunk_size)
+
+    p = pyaudio.PyAudio()
+
+    stream = p.open(format=FORMAT, channels=CHANNELS, rate=RATE, input=True, frames_per_buffer=CHUNK)
+
+    while True:
+        data = stream.read(CHUNK)
+        voices.put(data)
+        await asyncio.sleep(0.01)
+
+async def ws_send(websocket):
+    global voices
+    print("Started sending data!")
+    data_head = {
+        'vad_need': args.vad_needed,
+        'state': ''
+    }
+    await websocket.send(json.dumps(data_head))
+
+    while True:
+        while not voices.empty():
+            data = voices.get()
+            voices.task_done()
+            try:
+                await websocket.send(data)
+            except Exception as e:
+                print('Exception occurred:', e)
+                return  # Return to attempt reconnection
+            await asyncio.sleep(0.01)
+
+async def message(websocket):
+    while True:
+        try:
+            print(await websocket.recv())
+        except Exception as e:
+            print("Exception:", e)
+            return  # Return to attempt reconnection
+
+async def ws_client():
+    uri = "ws://{}:{}".format(args.host, args.port)
+    while True:
+        try:
+            async with websockets.connect(uri, subprotocols=["binary"], ping_interval=None) as websocket:
+                task1 = asyncio.create_task(record())
+                task2 = asyncio.create_task(ws_send(websocket))
+                task3 = asyncio.create_task(message(websocket))
+                await asyncio.gather(task1, task2, task3)
+        except Exception as e:
+            print("WebSocket connection failed: ", e)
+            await asyncio.sleep(5)  # Wait for 5 seconds before trying to reconnect
+
+asyncio.get_event_loop().run_until_complete(ws_client())

asr/funasr/ASR_server.py

@@ -0,0 +1,87 @@
+import asyncio
+import websockets
+import argparse
+import json
+from funasr import AutoModel
+import os
+
+# 设置日志级别
+import logging
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.CRITICAL)
+
+# 解析命令行参数
+parser = argparse.ArgumentParser()
+parser.add_argument("--host", type=str, default="0.0.0.0", help="host ip, localhost, 0.0.0.0")
+parser.add_argument("--port", type=int, default=10197, help="grpc server port")
+parser.add_argument("--ngpu", type=int, default=1, help="0 for cpu, 1 for gpu")
+args = parser.parse_args()
+
+# 初始化模型
+print("model loading")
+asr_model = AutoModel(model="paraformer-zh", model_revision="v2.0.4",
+                      vad_model="fsmn-vad", vad_model_revision="v2.0.4",
+                      punc_model="ct-punc-c", punc_model_revision="v2.0.4")
+print("model loaded")
+websocket_users = {}
+task_queue = asyncio.Queue()
+
+async def ws_serve(websocket, path):
+    global websocket_users
+    user_id = id(websocket)
+    websocket_users[user_id] = websocket
+    try:
+        async for message in websocket:
+            if isinstance(message, str):
+                data = json.loads(message)
+                if 'url' in data:
+                    await task_queue.put((websocket, data['url']))
+    except websockets.exceptions.ConnectionClosed as e:
+        logger.info(f"Connection closed: {e.reason}")
+    except Exception as e:
+        logger.error(f"Unexpected error: {e}")
+    finally:
+        logger.info(f"Cleaning up connection for user {user_id}")
+        if user_id in websocket_users:
+            del websocket_users[user_id]
+        await websocket.close()
+        logger.info("WebSocket closed")
+
+async def worker():
+    while True:
+        websocket, url = await task_queue.get()
+        if websocket.open:
+            await process_wav_file(websocket, url)
+        else:
+            logger.info("WebSocket connection is already closed when trying to process file")
+        task_queue.task_done()
+
+async def process_wav_file(websocket, url):
+    #热词
+    param_dict = {"sentence_timestamp": False}
+    with open("data/hotword.txt", "r", encoding="utf-8") as f:
+        lines = f.readlines()
+        lines = [line.strip() for line in lines]
+    hotword = " ".join(lines)
+    print(f"热词：{hotword}")
+    param_dict["hotword"] = hotword
+    wav_path = url
+    try:
+        res = asr_model.generate(input=wav_path,is_final=True, **param_dict)
+        if res:
+            if 'text' in res[0] and websocket.open:
+                await websocket.send(res[0]['text'])
+    except Exception as e:
+        print(f"Error during model.generate: {e}")
+    finally:
+        if os.path.exists(wav_path):
+            os.remove(wav_path)
+
+async def main():
+    start_server = websockets.serve(ws_serve, args.host, args.port, subprotocols=["binary"], ping_interval=10)
+    await start_server
+    worker_task = asyncio.create_task(worker())
+    await worker_task
+
+# 使用 asyncio 运行主函数
+asyncio.run(main())

asr/funasr/README.md

@@ -0,0 +1,32 @@
+##  语音服务介绍
+
+该服务以modelscope funasr语音识别为基础
+
+
+## Install
+pip install torch
+pip install modelscope
+pip install testresources
+pip install websockets
+pip install torchaudio
+pip install FunASR
+
+## Start server
+
+2、python -u ASR_server.py --host "0.0.0.0" --port 10197 --ngpu 0 
+
+## Fay connect
+更改fay/system.conf配置项，并重新启动fay.
+
+https://www.bilibili.com/video/BV1qs4y1g74e/?share_source=copy_web&vd_source=64cd9062f5046acba398177b62bea9ad
+
+
+## Acknowledge
+感谢
+1. 中科大脑算法工程师张聪聪
+2.  [cgisky1980](https://github.com/cgisky1980/FunASR) 
+3. [modelscope](https://github.com/modelscope/modelscope)
+4. [FunASR](https://github.com/alibaba-damo-academy/FunASR)
+5. [Fay数字人助理](https://github.com/TheRamU/Fay).
+
+--------------------------------------------------------------------------------------

asr/funasr/funasr_client_api.py

@@ -0,0 +1,178 @@
+'''
+  Copyright FunASR (https://github.com/alibaba-damo-academy/FunASR). All Rights
+  Reserved. MIT License  (https://opensource.org/licenses/MIT)
+  
+  2022-2023 by zhaomingwork@qq.com  
+'''
+# pip install websocket-client
+import ssl
+from websocket import ABNF
+from websocket import create_connection
+from queue import Queue
+import threading
+import traceback
+import json
+import time
+import numpy as np
+
+import pyaudio
+import asyncio
+import argparse
+
+# class for recognizer in websocket
+class Funasr_websocket_recognizer():
+    '''
+    python asr recognizer lib
+
+    '''
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--host", type=str, default="127.0.0.1", required=False, help="host ip, localhost, 0.0.0.0")
+    parser.add_argument("--port", type=int, default=10194, required=False, help="grpc server port")
+    parser.add_argument("--chunk_size", type=int, default=160, help="ms")
+    parser.add_argument("--vad_needed", type=bool, default=True)
+    args = parser.parse_args()
+
+    def __init__(self, host="127.0.0.1",
+                 port="10197",
+                 is_ssl=True,
+                 chunk_size="0, 10, 5",
+                 chunk_interval=10,
+                 mode="2pass",
+                 wav_name="default"):
+      '''
+          host: server host ip
+          port: server port
+          is_ssl: True for wss protocal, False for ws
+      '''
+      try:
+        if is_ssl == True:
+            ssl_context = ssl.SSLContext()
+            ssl_context.check_hostname = False
+            ssl_context.verify_mode = ssl.CERT_NONE
+            uri = "wss://{}:{}".format(host, port)
+            ssl_opt={"cert_reqs": ssl.CERT_NONE}
+        else:
+            uri = "ws://{}:{}".format(host, port)
+            ssl_context = None
+            ssl_opt=None
+        self.host = host
+        self.port = port
+ 
+        self.msg_queue = Queue() # used for recognized result text
+
+        print("connect to url",uri)
+        self.websocket=create_connection(uri, ssl=ssl_context, sslopt=ssl_opt)
+ 
+        self.thread_msg = threading.Thread(target=Funasr_websocket_recognizer.thread_rec_msg, args=(self,))
+        self.thread_msg.start()
+        chunk_size = [int(x) for x in  chunk_size.split(",")]
+        stride = int(60 *  chunk_size[1] / chunk_interval / 1000 * 16000 * 2)
+        chunk_num = (len(audio_bytes) - 1) // stride + 1
+       
+        message = json.dumps({"mode": mode,
+                              "chunk_size": chunk_size,
+                              "encoder_chunk_look_back": 4,
+                              "decoder_chunk_look_back": 1,
+                              "chunk_interval": chunk_interval,
+                              "wav_name": wav_name,
+                              "is_speaking": True})
+ 
+        self.websocket.send(message)
+ 
+        print("send json",message)
+      
+      except Exception as e:
+            print("Exception:", e)
+            traceback.print_exc()
+    
+    # async def record():
+    #     global voices 
+    #     FORMAT = pyaudio.paInt16
+    #     CHANNELS = 1
+    #     RATE = 16000
+    #     CHUNK = int(RATE / 1000 * args.chunk_size)
+
+    #     p = pyaudio.PyAudio()
+
+    #     stream = p.open(format=FORMAT, channels=CHANNELS, rate=RATE, input=True, frames_per_buffer=CHUNK)
+
+    #     while True:
+    #         data = stream.read(CHUNK)
+    #         voices.put(data)
+    #         await asyncio.sleep(0.01)
+
+
+    # threads for rev msg
+    def thread_rec_msg(self):
+        try:
+         while(True):
+           msg=self.websocket.recv()
+           if msg is None or len(msg) == 0:
+             continue
+           msg = json.loads(msg)
+           
+           self.msg_queue.put(msg)
+        except Exception as e:
+            print("client closed")
+ 
+    # feed data to asr engine, wait_time means waiting for result until time out
+    def feed_chunk(self, chunk, wait_time=0.01):
+        try:
+            self.websocket.send(chunk,  ABNF.OPCODE_BINARY)
+            # loop to check if there is a message, timeout in 0.01s
+            while(True):
+               msg = self.msg_queue.get(timeout=wait_time)
+               if self.msg_queue.empty():
+                  break
+                  
+            return msg
+        except:
+            return ""
+        
+    def close(self,timeout=1):
+        message = json.dumps({"is_speaking": False})
+        self.websocket.send(message)
+        # sleep for timeout seconds to wait for result
+        time.sleep(timeout)
+        msg=""
+        while(not self.msg_queue.empty()):
+            msg = self.msg_queue.get()
+        
+        self.websocket.close()
+        # only resturn the last msg
+        return msg
+        
+if __name__ == '__main__':
+    
+    print('example for Funasr_websocket_recognizer') 
+    import wave
+    wav_path = "long.wav"
+    # wav_path = "/Users/zhifu/Downloads/modelscope_models/speech_seaco_paraformer_large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/example/asr_example.wav"
+    with wave.open(wav_path, "rb") as wav_file:
+                params = wav_file.getparams()
+                frames = wav_file.readframes(wav_file.getnframes())
+                audio_bytes = bytes(frames)
+    
+ 
+    stride = int(60 * 10 / 10 / 1000 * 16000 * 2)
+    chunk_num = (len(audio_bytes) - 1) // stride + 1
+    # create an recognizer 
+    rcg = Funasr_websocket_recognizer()
+    # loop to send chunk
+    for i in range(chunk_num):
+
+            beg = i * stride
+            data = audio_bytes[beg:beg + stride]
+ 
+            text = rcg.feed_chunk(data,wait_time=0.02)
+            if len(text)>0:
+               print("text",text)
+            time.sleep(0.05)
+ 
+    # get last message
+    text = rcg.close(timeout=3)
+    print("text",text)
+ 
+    
+            

config.json

@@ -0,0 +1,44 @@
+{
+    "attribute": {
+        "age": "\u6210\u5e74",
+        "birth": "Github",
+        "constellation": "\u6c34\u74f6\u5ea7",
+        "contact": "qq467665317",
+        "gender": "\u5973",
+        "hobby": "\u53d1\u5446",
+        "job": "\u52a9\u7406",
+        "name": "\u83f2\u83f2",
+        "voice": "\u6653\u6653(azure)",
+        "zodiac": "\u86c7"
+    },
+    "interact": {
+        "QnA": "",
+        "maxInteractTime": 15,
+        "perception": {
+            "chat": 10,
+            "follow": 10,
+            "gift": 10,
+            "indifferent": 10,
+            "join": 10
+        },
+        "playSound": true,
+        "visualization": false
+    },
+    "items": [],
+    "source": {
+        "automatic_player_status": false,
+        "automatic_player_url": "http://127.0.0.1:6000",
+        "liveRoom": {
+            "enabled": true,
+            "url": ""
+        },
+        "record": {
+            "channels": 0,
+            "device": "",
+            "enabled": true
+        },
+        "wake_word": "\u4f60\u597d",
+        "wake_word_enabled": true,
+        "wake_word_type": "front"
+    }
+}

core/authorize_tb.py

@@ -0,0 +1,65 @@
+import sqlite3
+import time
+import threading
+import functools
+def synchronized(func):
+  @functools.wraps(func)
+  def wrapper(self, *args, **kwargs):
+    with self.lock:
+      return func(self, *args, **kwargs)
+  return wrapper
+class Authorize_Tb:
+
+    def __init__(self) -> None:
+        self.lock = threading.Lock()
+           
+   
+
+    #初始化
+    def init_tb(self):
+        conn = sqlite3.connect('fay.db')
+        c = conn.cursor()
+        c.execute('''
+            CREATE TABLE IF NOT EXISTS T_Authorize
+            (id INTEGER PRIMARY KEY     autoincrement,
+            userid        char(100),
+            accesstoken           TEXT,
+            expirestime           BigInt,
+            createtime         Int);
+        ''')
+        conn.commit()
+        conn.close()
+
+    #添加
+    @synchronized
+    def add(self,userid,accesstoken,expirestime):
+        self.init_tb()
+        conn = sqlite3.connect("fay.db")
+        cur = conn.cursor()
+        cur.execute("insert into T_Authorize (userid,accesstoken,expirestime,createtime) values (?,?,?,?)",(userid,accesstoken,expirestime,int(time.time())))
+        
+        conn.commit()
+        conn.close()
+        return cur.lastrowid   
+
+    #查询
+    @synchronized
+    def find_by_userid(self,userid):
+        self.init_tb()
+        conn = sqlite3.connect("fay.db")
+        cur = conn.cursor()
+        cur.execute("select accesstoken,expirestime from T_Authorize where userid = ? order by id desc limit 1",(userid,))
+        info = cur.fetchone()
+        conn.close()
+        return info
+
+    # 更新token
+    @synchronized
+    def update_by_userid(self, userid, new_accesstoken, new_expirestime):
+        self.init_tb()
+        conn = sqlite3.connect("fay.db")
+        cur = conn.cursor()
+        cur.execute("UPDATE T_Authorize SET accesstoken = ?, expirestime = ? WHERE userid = ?", 
+                    (new_accesstoken, new_expirestime, userid))
+        conn.commit()
+        conn.close()

core/content_db.py

@@ -0,0 +1,90 @@
+import sqlite3
+import time
+import threading
+import functools
+from utils import util
+def synchronized(func):
+  @functools.wraps(func)
+  def wrapper(self, *args, **kwargs):
+    with self.lock:
+      return func(self, *args, **kwargs)
+  return wrapper
+
+__content_tb = None
+def new_instance():
+    global __content_tb
+    if __content_tb is None:
+        __content_tb = Content_Db()
+        __content_tb.init_db()
+    return __content_tb
+
+
+class Content_Db:
+
+    def __init__(self) -> None:
+        self.lock = threading.Lock()
+           
+   
+
+    #初始化
+    def init_db(self):
+        conn = sqlite3.connect('fay.db')
+        c = conn.cursor()
+        c.execute('''CREATE TABLE IF NOT EXISTS T_Msg
+            (id INTEGER PRIMARY KEY     autoincrement,
+            type        char(10),
+            way        char(10),
+            content           TEXT    NOT NULL,
+            createtime         Int,
+            username TEXT DEFAULT 'User',
+            uid         Int);''')
+        conn.commit()
+        conn.close()
+       
+ 
+
+
+    #添加对话
+    @synchronized
+    def add_content(self,type,way,content,username='User',uid=0):
+        conn = sqlite3.connect("fay.db")
+        cur = conn.cursor()
+        try:
+            cur.execute("insert into T_Msg (type,way,content,createtime,username,uid) values (?,?,?,?,?,?)",(type,way,content,int(time.time()),username,uid))
+            conn.commit()
+        except:
+               util.log(1, "请检查参数是否有误")
+               conn.close()
+               return 0
+        conn.close()
+        return cur.lastrowid
+     
+        
+
+    #获取对话内容
+    @synchronized
+    def get_list(self,way,order,limit,uid=0):
+        conn = sqlite3.connect("fay.db")
+        cur = conn.cursor()
+        where_uid = ""
+        if int(uid) != 0:
+            where_uid = f" and uid = {uid} "
+        if(way == 'all'):
+            cur.execute("select type,way,content,createtime,datetime(createtime, 'unixepoch', 'localtime') as timetext,username from T_Msg  where 1 "+where_uid+" order by id "+order+" limit ?",(limit,))
+        elif(way == 'notappended'):
+            cur.execute("select type,way,content,createtime,datetime(createtime, 'unixepoch', 'localtime') as timetext,username from T_Msg where way != 'appended' "+where_uid+" order by id "+order+" limit ?",(limit,))
+        else:
+            cur.execute("select type,way,content,createtime,datetime(createtime, 'unixepoch', 'localtime') as timetext,username from T_Msg where way = ? "+where_uid+" order by id "+order+" limit ?",(way,limit,))
+
+        list = cur.fetchall()
+        conn.close()
+        return list
+
+
+
+
+
+   
+
+
+

core/fay_core.py

@@ -0,0 +1,497 @@
+#作用是处理交互逻辑，文字输入，语音、文字及情绪的发送、播放及展示输出
+import math
+import os
+import time
+import socket
+import wave
+import pygame
+import requests
+from pydub import AudioSegment
+
+# 适应模型使用
+import numpy as np
+import fay_booter
+from ai_module import baidu_emotion
+from core import wsa_server
+from core.interact import Interact
+from tts.tts_voice import EnumVoice
+from scheduler.thread_manager import MyThread
+from tts import tts_voice
+from utils import util, config_util
+from core import qa_service
+from utils import config_util as cfg
+from core import content_db
+from ai_module import nlp_cemotion
+from llm import nlp_rasa
+from llm import nlp_gpt
+from ai_module import yolov8
+from llm import nlp_VisualGLM
+from llm import nlp_lingju
+from llm import nlp_xingchen
+from llm import nlp_langchain
+from llm import nlp_ollama_api
+from llm import nlp_coze
+from core import member_db
+import threading
+import functools
+
+#加载配置
+cfg.load_config()
+if cfg.tts_module =='ali':
+    from tts.ali_tss import Speech
+elif cfg.tts_module == 'gptsovits':
+    from tts.gptsovits import Speech
+elif cfg.tts_module == 'gptsovits_v3':
+    from tts.gptsovits_v3 import Speech    
+elif cfg.tts_module == 'volcano':
+    from tts.volcano_tts import Speech
+else:
+    from tts.ms_tts_sdk import Speech
+
+#windows运行推送唇形数据
+import platform
+if platform.system() == "Windows":
+    import sys
+    sys.path.append("test/ovr_lipsync")
+    from test_olipsync import LipSyncGenerator
+    
+modules = {
+    "nlp_gpt": nlp_gpt,
+    "nlp_rasa": nlp_rasa,
+    "nlp_VisualGLM": nlp_VisualGLM,
+    "nlp_lingju": nlp_lingju,
+    "nlp_xingchen": nlp_xingchen,
+    "nlp_langchain": nlp_langchain,
+    "nlp_ollama_api": nlp_ollama_api,
+    "nlp_coze": nlp_coze
+
+}
+
+#大语言模型回复
+def handle_chat_message(msg, username='User'):
+    text = ''
+    textlist = []
+    try:
+        util.printInfo(1, username, '自然语言处理...')
+        tm = time.time()
+        cfg.load_config()
+        module_name = "nlp_" + cfg.key_chat_module
+        selected_module = modules.get(module_name)
+        if selected_module is None:
+            raise RuntimeError('请选择正确的nlp模型')   
+        if cfg.key_chat_module == 'rasa':
+            textlist = selected_module.question(msg)
+            text = textlist[0]['text'] 
+        else:
+            uid = member_db.new_instance().find_user(username)
+            text = selected_module.question(msg, uid)  
+        util.printInfo(1, username, '自然语言处理完成. 耗时: {} ms'.format(math.floor((time.time() - tm) * 1000)))
+        if text == '哎呀，你这么说我也不懂，详细点呗' or text == '':
+            util.printInfo(1, username, '[!] 自然语言无语了！')
+            text = '哎呀，你这么说我也不懂，详细点呗'  
+    except BaseException as e:
+        print(e)
+        util.printInfo(1, username, '自然语言处理错误！')
+        text = '哎呀，你这么说我也不懂，详细点呗'   
+
+    return text,textlist
+
+#可以使用自动播放的标记    
+can_auto_play = True
+auto_play_lock = threading.Lock()
+
+class FeiFei:
+    def __init__(self):
+        self.lock = threading.Lock()
+        self.mood = 0.0  # 情绪值
+        self.old_mood = 0.0
+        self.item_index = 0
+        self.X = np.array([1, 0, 0, 0, 0, 0, 0, 0]).reshape(1, -1)  # 适应模型变量矩阵
+        # self.W = np.array([0.01577594,1.16119452,0.75828,0.207746,1.25017864,0.1044121,0.4294899,0.2770932]).reshape(-1,1) #适应模型变量矩阵
+        self.W = np.array([0.0, 0.6, 0.1, 0.7, 0.3, 0.0, 0.0, 0.0]).reshape(-1, 1)  # 适应模型变量矩阵
+
+        self.wsParam = None
+        self.wss = None
+        self.sp = Speech()
+        self.speaking = False #声音是否在播放
+        self.__running = True
+        self.sp.connect()  #TODO 预连接
+        self.cemotion = None
+
+    #语音消息处理检查是否命中q&a
+    def __get_answer(self, interleaver, text):
+        answer = None
+        # 全局问答
+        answer = qa_service.QAService().question('qa',text)
+        if answer is not None:
+            return answer
+        
+       
+    #语音消息处理
+    def __process_interact(self, interact: Interact):
+        if self.__running:
+            try:
+                index = interact.interact_type
+                if index == 1: #语音文字交互
+                    #记录用户问题,方便obs等调用
+                    self.write_to_file("./logs", "asr_result.txt",  interact.data["msg"])
+
+                    #同步用户问题到数字人
+                    if wsa_server.get_instance().is_connected(interact.data.get("user")): 
+                        content = {'Topic': 'Unreal', 'Data': {'Key': 'question', 'Value': interact.data["msg"]}, 'Username' : interact.data.get("user")}
+                        wsa_server.get_instance().add_cmd(content)
+
+                    #记录用户
+                    username = interact.data.get("user", "User")
+                    if member_db.new_instance().is_username_exist(username)  == "notexists":
+                        member_db.new_instance().add_user(username)
+                    uid = member_db.new_instance().find_user(username)
+
+                    #记录用户问题
+                    content_db.new_instance().add_content('member','speak',interact.data["msg"], username, uid)
+                    if wsa_server.get_web_instance().is_connected(username):
+                        wsa_server.get_web_instance().add_cmd({"panelReply": {"type":"member","content":interact.data["msg"], "username":username, "uid":uid}, "Username" : username})
+                    
+                    #确定是否命中q&a
+                    answer = self.__get_answer(interact.interleaver, interact.data["msg"])
+
+                    #大语言模型回复    
+                    text = ''
+                    textlist = []
+                    if answer is None:
+                        if wsa_server.get_web_instance().is_connected(username):
+                            wsa_server.get_web_instance().add_cmd({"panelMsg": "思考中...", "Username" : username, 'robot': f'http://{cfg.fay_url}:5000/robot/Thinking.jpg'})
+                        if wsa_server.get_instance().is_connected(username):
+                            content = {'Topic': 'Unreal', 'Data': {'Key': 'log', 'Value': "思考中..."}, 'Username' : username, 'robot': f'http://{cfg.fay_url}:5000/robot/Thinking.jpg'}
+                            wsa_server.get_instance().add_cmd(content)
+                        text,textlist = handle_chat_message(interact.data["msg"], username)
+
+                        qa_service.QAService().record_qapair(interact.data["msg"], text)#沟通记录缓存到qa文件
+                    else: 
+                        text = answer
+
+                    #记录回复    
+                    self.write_to_file("./logs", "answer_result.txt", text)
+                    content_db.new_instance().add_content('fay','speak',text, username, uid)
+
+                    #文字输出：面板、聊天窗、log、数字人
+                    if wsa_server.get_web_instance().is_connected(username):
+                        wsa_server.get_web_instance().add_cmd({"panelMsg": text, "Username" : username, 'robot': f'http://{cfg.fay_url}:5000/robot/Speaking.jpg'})
+                        wsa_server.get_web_instance().add_cmd({"panelReply": {"type":"fay","content":text, "username":username, "uid":uid}, "Username" : username})
+                    if len(textlist) > 1:
+                        i = 1
+                        while i < len(textlist):
+                            content_db.new_instance().add_content('fay','speak',textlist[i]['text'], username, uid)
+                            if wsa_server.get_web_instance().is_connected(username):
+                                wsa_server.get_web_instance().add_cmd({"panelReply": {"type":"fay","content":textlist[i]['text'], "username":username, "uid":uid}, "Username" : username, 'robot': f'http://{cfg.fay_url}:5000/robot/Speaking.jpg'})
+                            i+= 1                                
+                    util.printInfo(1, interact.data.get('user'), '({}) {}'.format(self.__get_mood_voice(), text))                            
+                    if wsa_server.get_instance().is_connected(username):
+                        content = {'Topic': 'Unreal', 'Data': {'Key': 'text', 'Value': text}, 'Username' : username, 'robot': f'http://{cfg.fay_url}:5000/robot/Speaking.jpg'}
+                        wsa_server.get_instance().add_cmd(content)
+                
+                    #声音输出
+                    MyThread(target=self.say, args=[interact, text]).start()
+                    
+                    return text      
+                
+                elif (index == 2):#透传模式，用于适配自动播放控制及agent的通知工具
+                    #记录用户
+                    username = interact.data.get("user", "User")
+                    if member_db.new_instance().is_username_exist(username)  == "notexists":
+                        member_db.new_instance().add_user(username)
+                    uid = member_db.new_instance().find_user(username)
+                    
+                    #TODO 这里可以通过qa来触发指定的脚本操作，如ppt翻页等
+
+                    if interact.data.get("text"):
+                        #记录回复
+                        text = interact.data.get("text")
+                        self.write_to_file("./logs", "answer_result.txt", text)
+                        content_db.new_instance().add_content('fay','speak', text, username, uid)
+
+                        #文字输出：面板、聊天窗、log、数字人
+                        if wsa_server.get_web_instance().is_connected(username):
+                            wsa_server.get_web_instance().add_cmd({"panelMsg": text, "Username" : username, 'robot': f'http://{cfg.fay_url}:5000/robot/Speaking.jpg'})
+                            wsa_server.get_web_instance().add_cmd({"panelReply": {"type":"fay","content":text, "username":username, "uid":uid}, "Username" : username})  
+                        util.printInfo(1, interact.data.get('user'), '({}) {}'.format(self.__get_mood_voice(), text))                            
+                        if wsa_server.get_instance().is_connected(username):
+                            content = {'Topic': 'Unreal', 'Data': {'Key': 'text', 'Value': text}, 'Username' : username, 'robot': f'http://{cfg.fay_url}:5000/robot/Speaking.jpg'}
+                            wsa_server.get_instance().add_cmd(content)
+                    
+                    #声音输出
+                    MyThread(target=self.say, args=[interact, text]).start()
+   
+            except BaseException as e:
+                print(e)
+                return e
+        else:
+            return "还没有开始运行"
+
+    #记录问答到log
+    def write_to_file(self, path, filename, content):
+        if not os.path.exists(path):
+            os.makedirs(path)
+        full_path = os.path.join(path, filename)
+        with open(full_path, 'w', encoding='utf-8') as file:
+            file.write(content)
+            file.flush()  
+            os.fsync(file.fileno()) 
+
+    #触发语音交互
+    def on_interact(self, interact: Interact):
+        MyThread(target=self.__update_mood, args=[interact]).start()
+        return self.__process_interact(interact)
+
+    # 发送情绪
+    def __send_mood(self):
+         while self.__running:
+            time.sleep(3)
+            if wsa_server.get_instance().is_connected("User"):
+                if  self.old_mood != self.mood:
+                    content = {'Topic': 'Unreal', 'Data': {'Key': 'mood', 'Value': self.mood}}
+                    wsa_server.get_instance().add_cmd(content)
+                    self.old_mood = self.mood
+
+    #TODO 考虑重构这个逻辑  
+    # 更新情绪
+    def __update_mood(self, interact):
+        perception = config_util.config["interact"]["perception"]
+        if interact.interact_type == 1:
+            try:
+                if cfg.ltp_mode == "cemotion":
+                    result = nlp_cemotion.get_sentiment(self.cemotion, interact.data["msg"])
+                    chat_perception = perception["chat"]
+                    if result >= 0.5 and result <= 1:
+                       self.mood = self.mood + (chat_perception / 150.0)
+                    elif result <= 0.2:
+                       self.mood = self.mood - (chat_perception / 100.0)
+                else:
+                    if str(cfg.baidu_emotion_api_key) == '' or str(cfg.baidu_emotion_app_id) == '' or str(cfg.baidu_emotion_secret_key) == '':
+                        self.mood = 0
+                    else:
+                        result = int(baidu_emotion.get_sentiment(interact.data["msg"]))
+                        chat_perception = perception["chat"]
+                        if result >= 2:
+                            self.mood = self.mood + (chat_perception / 150.0)
+                        elif result == 0:
+                            self.mood = self.mood - (chat_perception / 100.0)
+            except BaseException as e:
+                self.mood = 0
+                print("[System] 情绪更新错误！")
+                print(e)
+
+        elif interact.interact_type == 2:
+            self.mood = self.mood + (perception["join"] / 100.0)
+
+        elif interact.interact_type == 3:
+            self.mood = self.mood + (perception["gift"] / 100.0)
+
+        elif interact.interact_type == 4:
+            self.mood = self.mood + (perception["follow"] / 100.0)
+
+        if self.mood >= 1:
+            self.mood = 1
+        if self.mood <= -1:
+            self.mood = -1
+
+    #获取不同情绪声音
+    def __get_mood_voice(self):
+        voice = tts_voice.get_voice_of(config_util.config["attribute"]["voice"])
+        if voice is None:
+            voice = EnumVoice.XIAO_XIAO
+        styleList = voice.value["styleList"]
+        sayType = styleList["calm"]
+        if -1 <= self.mood < -0.5:
+            sayType = styleList["angry"]
+        if -0.5 <= self.mood < -0.1:
+            sayType = styleList["lyrical"]
+        if -0.1 <= self.mood < 0.1:
+            sayType = styleList["calm"]
+        if 0.1 <= self.mood < 0.5:
+            sayType = styleList["assistant"]
+        if 0.5 <= self.mood <= 1:
+            sayType = styleList["cheerful"]
+        return sayType
+
+    # 合成声音
+    def say(self, interact, text):
+        try:
+            result = None
+            audio_url = interact.data.get('audio')#透传的音频
+            if audio_url is not None:
+                file_name = 'sample-' + str(int(time.time() * 1000)) + '.wav'
+                result = self.download_wav(audio_url, './samples/', file_name)
+            elif config_util.config["interact"]["playSound"] or wsa_server.get_instance().is_connected(interact.data.get("user")) or self.__is_send_remote_device_audio(interact):#tts
+                util.printInfo(1,  interact.data.get('user'), '合成音频...')
+                tm = time.time()
+                result = self.sp.to_sample(text.replace("*", ""), self.__get_mood_voice())
+                util.printInfo(1,  interact.data.get('user'), '合成音频完成. 耗时: {} ms 文件:{}'.format(math.floor((time.time() - tm) * 1000), result))
+
+            if result is not None:            
+                MyThread(target=self.__process_output_audio, args=[result, interact, text]).start()
+                return result
+            else:
+                if wsa_server.get_web_instance().is_connected(interact.data.get('user')):
+                    wsa_server.get_web_instance().add_cmd({"panelMsg": "", 'Username' : interact.data.get('user'), 'robot': f'http://{cfg.fay_url}:5000/robot/Normal.jpg'})
+                if wsa_server.get_instance().is_connected(interact.data.get("user")):
+                    content = {'Topic': 'Unreal', 'Data': {'Key': 'log', 'Value': ''}, 'Username' : interact.data.get('user'), 'robot': f'http://{cfg.fay_url}:5000/robot/Normal.jpg'}
+                    wsa_server.get_instance().add_cmd(content)               
+                
+        except BaseException as e:
+            print(e)
+        return None
+    
+    #下载wav
+    def download_wav(self, url, save_directory, filename):
+        try:
+            # 发送HTTP GET请求以获取WAV文件内容
+            response = requests.get(url, stream=True)
+            response.raise_for_status()  # 检查请求是否成功
+
+            # 确保保存目录存在
+            if not os.path.exists(save_directory):
+                os.makedirs(save_directory)
+
+            # 构建保存文件的路径
+            save_path = os.path.join(save_directory, filename)
+
+            # 将WAV文件内容保存到指定文件
+            with open(save_path, 'wb') as f:
+                for chunk in response.iter_content(chunk_size=1024):
+                    if chunk:
+                        f.write(chunk)
+
+            return save_path
+        except requests.exceptions.RequestException as e:
+            print(f"[Error] Failed to download file: {e}")
+            return None
+
+
+    #面板播放声音
+    def __play_sound(self, file_url, audio_length, interact):
+        util.printInfo(1,  interact.data.get('user'), '播放音频...')
+        pygame.mixer.init()
+        pygame.mixer.music.load(file_url)
+        pygame.mixer.music.play()
+
+        #等待音频播放完成，唤醒模式不用等待        
+        length = 0
+        while True:
+            if audio_length + 0.01 > length:
+                length = length + 0.01
+                time.sleep(0.01)
+            else:
+                break
+        if wsa_server.get_instance().is_connected(interact.data.get("user")):
+            wsa_server.get_web_instance().add_cmd({"panelMsg": "", 'Username' : interact.data.get('user')})
+    
+    #推送远程音频
+    def __send_remote_device_audio(self, file_url, interact):
+        delkey = None    
+        for key, value in fay_booter.DeviceInputListenerDict.items():
+            if value.username == interact.data.get("user") and value.isOutput: #按username选择推送，booter.devicelistenerdice按用户名记录
+                try:
+                    value.deviceConnector.send(b"\x00\x01\x02\x03\x04\x05\x06\x07\x08") # 发送音频开始标志，同时也检查设备是否在线
+                    wavfile = open(os.path.abspath(file_url), "rb")
+                    data = wavfile.read(102400)
+                    total = 0
+                    while data:
+                        total += len(data)
+                        value.deviceConnector.send(data)
+                        data = wavfile.read(102400)
+                        time.sleep(0.0001)
+                    value.deviceConnector.send(b'\x08\x07\x06\x05\x04\x03\x02\x01\x00')# 发送音频结束标志
+                    util.printInfo(1, value.username, "远程音频发送完成：{}".format(total))
+                except socket.error as serr:
+                    util.printInfo(1, value.username, "远程音频输入输出设备已经断开：{}".format(key)) 
+                    value.stop()
+                    delkey = key
+        if delkey:
+             value =  fay_booter.DeviceInputListenerDict.pop(delkey)
+             if wsa_server.get_web_instance().is_connected(interact.data.get('user')):
+                wsa_server.get_web_instance().add_cmd({"remote_audio_connect": False, "Username" : interact.data.get('user')})
+
+    def __is_send_remote_device_audio(self, interact):
+        for key, value in fay_booter.DeviceInputListenerDict.items():
+            if value.username == interact.data.get("user") and value.isOutput:
+                return True
+        return False 
+
+    #输出音频处理
+    def __process_output_audio(self, file_url, interact, text):
+        try:
+            try:
+                audio = AudioSegment.from_wav(file_url)
+                audio_length = len(audio) / 1000.0  # 时长以秒为单位
+            except Exception as e:
+                audio_length = 3
+
+            #自动播放关闭
+            global auto_play_lock
+            global can_auto_play
+            with auto_play_lock:
+                can_auto_play = False
+
+            self.speaking = True
+            #推送远程音频
+            MyThread(target=self.__send_remote_device_audio, args=[file_url, interact]).start()       
+
+            #发送音频给数字人接口
+            if wsa_server.get_instance().is_connected(interact.data.get("user")):
+                content = {'Topic': 'Unreal', 'Data': {'Key': 'audio', 'Value': os.path.abspath(file_url), 'HttpValue': f'http://{cfg.fay_url}:5000/audio/' + os.path.basename(file_url),  'Text': text, 'Time': audio_length, 'Type': 'interact'}, 'Username' : interact.data.get('user')}
+                #计算lips
+                if platform.system() == "Windows":
+                    try:
+                        lip_sync_generator = LipSyncGenerator()
+                        viseme_list = lip_sync_generator.generate_visemes(os.path.abspath(file_url))
+                        consolidated_visemes = lip_sync_generator.consolidate_visemes(viseme_list)
+                        content["Data"]["Lips"] = consolidated_visemes
+                    except Exception as e:
+                        print(e)
+                        util.printInfo(1, interact.data.get("user"),  "唇型数据生成失败")
+                wsa_server.get_instance().add_cmd(content)
+                util.printInfo(1, interact.data.get("user"),  "数字人接口发送音频数据成功")
+            
+            #播放完成通知
+            threading.Timer(audio_length, self.send_play_end_msg, [interact]).start()
+
+            #面板播放
+            if config_util.config["interact"]["playSound"]:
+                 self.__play_sound(file_url, audio_length, interact)
+            
+        except Exception as e:
+            print(e)
+
+    def send_play_end_msg(self, interact):
+        if wsa_server.get_web_instance().is_connected(interact.data.get('user')):
+            wsa_server.get_web_instance().add_cmd({"panelMsg": "", 'Username' : interact.data.get('user'), 'robot': f'http://{cfg.fay_url}:5000/robot/Normal.jpg'})
+        if wsa_server.get_instance().is_connected(interact.data.get("user")):
+            content = {'Topic': 'Unreal', 'Data': {'Key': 'log', 'Value': ""}, 'Username' : interact.data.get('user'), 'robot': f'http://{cfg.fay_url}:5000/robot/Normal.jpg'}
+            wsa_server.get_instance().add_cmd(content)
+        if config_util.config["interact"]["playSound"]:
+            util.printInfo(1, interact.data.get('user'), '结束播放！')
+        
+        #恢复自动播放(如何有)
+        global auto_play_lock
+        global can_auto_play
+        with auto_play_lock:
+            can_auto_play = True
+        
+        self.speaking = False
+
+    #启动核心服务
+    def start(self):
+        if cfg.ltp_mode == "cemotion":
+            from cemotion import Cemotion
+            self.cemotion = Cemotion()
+        MyThread(target=self.__send_mood).start()
+
+    #停止核心服务
+    def stop(self):
+        self.__running = False
+        self.speaking = False
+        self.sp.close()
+        wsa_server.get_web_instance().add_cmd({"panelMsg": ""})
+        content = {'Topic': 'Unreal', 'Data': {'Key': 'log', 'Value': ""}}
+        wsa_server.get_instance().add_cmd(content)

core/interact.py

@@ -0,0 +1,7 @@
+#inerleaver:mic、text、socket、auto_play。interact_type:1、语音/文字交互；2、穿透。
+class Interact:
+
+    def __init__(self, interleaver: str, interact_type: int, data: dict):
+        self.interleaver = interleaver
+        self.interact_type = interact_type
+        self.data = data

core/member_db.py

@@ -0,0 +1,129 @@
+import sqlite3
+import time
+import threading
+import functools
+def synchronized(func):
+  @functools.wraps(func)
+  def wrapper(self, *args, **kwargs):
+    with self.lock:
+      return func(self, *args, **kwargs)
+  return wrapper
+
+__member_db = None
+def new_instance():
+    global __member_db
+    if __member_db is None:
+        __member_db = Member_Db()
+        __member_db.init_db()
+    return __member_db
+
+
+class Member_Db:
+
+    def __init__(self) -> None:
+        self.lock = threading.Lock()
+           
+   
+
+    #初始化
+    def init_db(self):
+        conn = sqlite3.connect('user_profiles.db')
+        c = conn.cursor()
+        c.execute('''CREATE TABLE IF NOT EXISTS T_Member
+            (id INTEGER PRIMARY KEY     autoincrement,
+            username        TEXT    NOT NULL UNIQUE);''')
+        conn.commit()
+        conn.close()
+       
+    # 添加新用户
+    @synchronized
+    def add_user(self, username):
+        if self.is_username_exist(username) == "notexists":
+            conn = sqlite3.connect('user_profiles.db')
+            c = conn.cursor()
+            c.execute('INSERT INTO T_Member (username) VALUES (?)', (username,))
+            conn.commit()
+            conn.close()
+            return "success"
+        else:
+           return f"Username '{username}' already exists."
+
+    # 修改用户名
+    @synchronized
+    def update_user(self, username, new_username):
+        if self.is_username_exist(new_username) == "notexists":
+            conn = sqlite3.connect('user_profiles.db')
+            c = conn.cursor()
+            c.execute('UPDATE T_Member SET username = ? WHERE username = ?', (new_username, username))
+            conn.commit()
+            conn.close()
+            return "success"
+        else:
+            return f"Username '{new_username}' already exists."
+
+    # 删除用户
+    @synchronized
+    def delete_user(self, username):
+        conn = sqlite3.connect('user_profiles.db')
+        c = conn.cursor()
+        c.execute('DELETE FROM T_Member WHERE username = ?', (username,))
+        conn.commit()
+        conn.close()
+        return "success"
+
+    # 检查用户名是否已存在
+    def is_username_exist(self, username):
+        conn = sqlite3.connect('user_profiles.db')
+        c = conn.cursor()
+        c.execute('SELECT COUNT(*) FROM T_Member WHERE username = ?', (username,))
+        result = c.fetchone()[0]
+        conn.close()
+        if result > 0:
+            return "exists"
+        else:
+            return "notexists"
+
+    def find_user(self, username):
+        conn = sqlite3.connect('user_profiles.db')
+        c = conn.cursor()
+        c.execute('SELECT * FROM T_Member WHERE username = ?', (username,))
+        result = c.fetchone()
+        conn.close()
+        if result is None:
+            return 0
+        else:
+           return result[0]
+
+    @synchronized
+    def query(self, sql):
+        try:
+            conn = sqlite3.connect('user_profiles.db')
+            c = conn.cursor()
+            c.execute(sql)
+            results = c.fetchall()
+            conn.commit()
+            conn.close()
+            return results
+        except Exception as e:
+            return f"执行时发生错误：{str(e)}"
+
+
+    # 获取所有用户
+    @synchronized
+    def get_all_users(self):
+        conn = sqlite3.connect('user_profiles.db')
+        c = conn.cursor()
+        c.execute('SELECT * FROM T_Member')
+        results = c.fetchall()
+        conn.close()
+        return results
+        
+
+
+
+
+
+   
+
+
+

core/qa_service.py

@@ -0,0 +1,101 @@
+import os
+import csv
+import difflib
+from utils import config_util as cfg
+from scheduler.thread_manager import MyThread
+import shlex
+import subprocess
+import time
+from utils import util
+
+class QAService:
+    
+    def __init__(self):
+         # 人设提问关键字
+        self.attribute_keyword = [
+            [['你叫什么名字', '你的名字是什么'], 'name'],
+            [['你是男的还是女的', '你是男生还是女生', '你的性别是什么', '你是男生吗', '你是女生吗', '你是男的吗', '你是女的吗', '你是男孩子吗', '你是女孩子吗', ], 'gender', ],
+            [['你今年多大了', '你多大了', '你今年多少岁', '你几岁了', '你今年几岁了', '你今年几岁了', '你什么时候出生', '你的生日是什么', '你的年龄'], 'age', ],
+            [['你的家乡在哪', '你的家乡是什么', '你家在哪', '你住在哪', '你出生在哪', '你的出生地在哪', '你的出生地是什么', ], 'birth', ],
+            [['你的生肖是什么', '你属什么', ], 'zodiac', ],
+            [['你是什么座', '你是什么星座', '你的星座是什么', ], 'constellation', ],
+            [['你是做什么的', '你的职业是什么', '你是干什么的', '你的职位是什么', '你的工作是什么', '你是做什么工作的'], 'job', ],
+            [['你的爱好是什么', '你有爱好吗', '你喜欢什么', '你喜欢做什么'], 'hobby'],
+            [['联系方式', '联系你们', '怎么联系客服', '有没有客服'], 'contact']
+        ]
+
+        self.command_keyword = [
+            [['关闭', '再见', '你走吧'], 'stop'],
+            [['静音', '闭嘴', '我想静静'], 'mute'],
+            [['取消静音', '你在哪呢', '你可以说话了'], 'unmute'],
+            [['换个性别', '换个声音'], 'changeVoice']
+        ]
+
+    def question(self, query_type, text):
+        if query_type == 'qa':
+            answer_dict = self.__read_qna(cfg.config['interact']['QnA'])
+            answer, action = self.__get_keyword(answer_dict, text, query_type)
+            if action:
+                MyThread(target=self.__run, args=[action]).start()
+            return answer
+    
+        elif query_type == 'Persona':
+            answer_dict = self.attribute_keyword
+            answer, action  = self.__get_keyword(answer_dict, text, query_type)
+        elif query_type == 'command':
+            answer, action  = self.__get_keyword(self.command_keyword, text, query_type)
+        return answer
+
+    def __run(self, action):
+        time.sleep(0.1)
+        args = shlex.split(action)  # 分割命令行参数
+        subprocess.Popen(args)
+
+    def __read_qna(self, filename):
+        qna = []
+        try:
+            with open(filename, 'r', encoding='utf-8') as csvfile:
+                reader = csv.reader(csvfile)
+                next(reader)  # 跳过表头
+                for row in reader:
+                    if len(row) >= 2:
+                        qna.append([row[0].split(";"), row[1], row[2] if len(row) >= 3 else None])
+        except Exception as e:
+            util.log(1, 'qa文件没有指定，不匹配qa')
+        return qna
+
+    def record_qapair(self, question, answer):
+        if not cfg.config['interact']['QnA'] or cfg.config['interact']['QnA'][-3:] != 'csv':
+            util.log(1, 'qa文件没有指定，不记录大模型回复')
+            return
+        log_file = cfg.config['interact']['QnA']  # 指定 CSV 文件的名称或路径
+        file_exists = os.path.isfile(log_file)
+        with open(log_file, 'a', newline='', encoding='utf-8') as csvfile:
+            writer = csv.writer(csvfile)
+            if not file_exists:
+                # 写入表头
+                writer.writerow(['Question', 'Answer'])
+            writer.writerow([question, answer])
+
+    def __get_keyword(self, keyword_dict, text, query_type):
+        last_similar = 0
+        last_answer = ''
+        last_action = ''
+        for qa in keyword_dict:
+            for quest in qa[0]:
+                similar = self.__string_similar(text, quest)
+                if quest in text:
+                    similar += 0.3
+                if similar > last_similar:
+                    last_similar = similar
+                    last_answer = qa[1]
+                    if query_type == "qa":
+                        last_action = qa[2]
+        if last_similar >= 0.6:
+            return last_answer, last_action
+        return None, None
+
+    def __string_similar(self, s1, s2):
+        return difflib.SequenceMatcher(None, s1, s2).quick_ratio()
+
+

core/recorder.py

@@ -0,0 +1,342 @@
+#作用是音频录制，对于aliyun asr来说，边录制边stt，但对于其他来说，是先保存成文件再推送给asr模型，通过实现子类的方式（fay_booter.py 上有实现）来管理音频流的来源
+import audioop
+import math
+import time
+import threading
+from abc import abstractmethod
+
+from asr.ali_nls import ALiNls
+from asr.funasr import FunASR
+from core import wsa_server
+from scheduler.thread_manager import MyThread
+from utils import util
+from utils import config_util as cfg
+import numpy as np
+import tempfile
+import wave
+from core import fay_core
+from core import interact
+# 启动时间 (秒)
+_ATTACK = 0.2
+
+# 释放时间 (秒)
+_RELEASE = 0.7
+
+
+class Recorder:
+
+    def __init__(self, fay):
+        self.__fay = fay
+        self.__running = True
+        self.__processing = False
+        self.__history_level = []
+        self.__history_data = []
+        self.__dynamic_threshold = 0.5 # 声音识别的音量阈值
+
+        self.__MAX_LEVEL = 25000
+        self.__MAX_BLOCK = 100
+        
+        #Edit by xszyou in 20230516:增加本地asr
+        self.ASRMode = cfg.ASR_mode
+        self.__aLiNls = None
+        self.is_awake = False
+        self.wakeup_matched = False
+        if cfg.config['source']['wake_word_enabled']:
+            self.timer = threading.Timer(60, self.reset_wakeup_status)  # 60秒后执行reset_wakeup_status方法
+        self.username = 'User' #默认用户，子类实现时会重写
+        self.channels = 1
+        self.sample_rate = 16000
+
+    def asrclient(self):
+        if self.ASRMode == "ali":
+            asrcli = ALiNls(self.username)
+        elif self.ASRMode == "funasr" or self.ASRMode == "sensevoice":
+            asrcli = FunASR(self.username)
+        return asrcli
+
+    def save_buffer_to_file(self, buffer):
+        temp_file = tempfile.NamedTemporaryFile(delete=False, suffix=".wav", dir="cache_data")
+        wf = wave.open(temp_file.name, 'wb')
+        wf.setnchannels(1)
+        wf.setsampwidth(2)  
+        wf.setframerate(16000)
+        wf.writeframes(buffer)
+        wf.close()
+        return temp_file.name
+
+    def __get_history_average(self, number):
+        total = 0
+        num = 0
+        for i in range(len(self.__history_level) - 1, -1, -1):
+            level = self.__history_level[i]
+            total += level
+            num += 1
+            if num >= number:
+                break
+        return total / num
+
+    def __get_history_percentage(self, number):
+        return (self.__get_history_average(number) / self.__MAX_LEVEL) * 1.05 + 0.02
+
+    def reset_wakeup_status(self):
+        self.wakeup_matched = False  
+        with fay_core.auto_play_lock:
+            fay_core.can_auto_play = True
+
+    def __waitingResult(self, iat: asrclient, audio_data):
+        self.processing = True
+        t = time.time()
+        tm = time.time()
+        if self.ASRMode == "funasr"  or self.ASRMode == "sensevoice":
+            file_url = self.save_buffer_to_file(audio_data)
+            self.__aLiNls.send_url(file_url)
+        
+        # return
+        # 等待结果返回
+        while not iat.done and time.time() - t < 1:
+            time.sleep(0.01)
+        text = iat.finalResults
+        util.printInfo(1, self.username, "语音处理完成！ 耗时: {} ms".format(math.floor((time.time() - tm) * 1000)))
+        if len(text) > 0:
+            if cfg.config['source']['wake_word_enabled']:
+                #普通唤醒模式
+                if cfg.config['source']['wake_word_type'] == 'common':
+
+                    if not self.wakeup_matched:
+                        #唤醒词判断
+                        wake_word =  cfg.config['source']['wake_word']
+                        wake_word_list = wake_word.split(',')
+                        wake_up = False
+                        for word in wake_word_list:
+                            if word in text:
+                                    wake_up = True
+                        if wake_up:
+                            util.printInfo(1, self.username, "唤醒成功！")
+                            if wsa_server.get_web_instance().is_connected(self.username):
+                                wsa_server.get_web_instance().add_cmd({"panelMsg": "唤醒成功！", "Username" : self.username , 'robot': f'http://{cfg.fay_url}:5000/robot/Listening.jpg'})
+                            if wsa_server.get_instance().is_connected(self.username):
+                                content = {'Topic': 'Unreal', 'Data': {'Key': 'log', 'Value': "唤醒成功！"}, 'Username' : self.username, 'robot': f'http://{cfg.fay_url}:5000/robot/Listening.jpg'}
+                                wsa_server.get_instance().add_cmd(content)
+                            self.wakeup_matched = True  # 唤醒成功
+                            with fay_core.auto_play_lock:
+                                fay_core.can_auto_play = False
+                            #self.on_speaking(text)
+                            intt = interact.Interact("auto_play", 2, {'user': self.username, 'text': "在呢，你说？"})
+                            self.__fay.on_interact(intt)
+                            self.processing = False
+                            self.timer.cancel()  # 取消之前的计时器任务
+                            
+                        else:
+                            util.printInfo(1, self.username, "[!] 待唤醒！")
+                            if wsa_server.get_web_instance().is_connected(self.username):
+                                wsa_server.get_web_instance().add_cmd({"panelMsg": "[!] 待唤醒！", "Username" : self.username , 'robot': f'http://{cfg.fay_url}:5000/robot/Normal.jpg'})
+                            if wsa_server.get_instance().is_connected(self.username):
+                                content = {'Topic': 'Unreal', 'Data': {'Key': 'log', 'Value': "[!] 待唤醒！"}, 'Username' : self.username, 'robot': f'http://{cfg.fay_url}:5000/robot/Normal.jpg'}
+                                wsa_server.get_instance().add_cmd(content)
+                    else:
+                        self.on_speaking(text)
+                        self.processing = False
+                        self.timer.cancel()  # 取消之前的计时器任务
+                        self.timer = threading.Timer(60, self.reset_wakeup_status)  # 重设计时器为60秒
+                        self.timer.start()
+                
+                #前置唤醒词模式
+                elif  cfg.config['source']['wake_word_type'] == 'front':
+                    wake_word =  cfg.config['source']['wake_word']
+                    wake_word_list = wake_word.split(',')
+                    wake_up = False
+                    for word in wake_word_list:
+                        if text.startswith(word):
+                            wake_up_word = word
+                            wake_up = True
+                            break
+                    if wake_up:
+                        util.printInfo(1, self.username, "唤醒成功！")
+                        if wsa_server.get_web_instance().is_connected(self.username):
+                            wsa_server.get_web_instance().add_cmd({"panelMsg": "唤醒成功！", "Username" : self.username , 'robot': f'http://{cfg.fay_url}:5000/robot/Listening.jpg'})
+                        if wsa_server.get_instance().is_connected(self.username):
+                            content = {'Topic': 'Unreal', 'Data': {'Key': 'log', 'Value': "唤醒成功！"}, 'Username' : self.username, 'robot': f'http://{cfg.fay_url}:5000/robot/Listening.jpg'}
+                            wsa_server.get_instance().add_cmd(content)
+                        #去除唤醒词后语句
+                        question = text#[len(wake_up_word):].lstrip()
+                        self.on_speaking(question)
+                        self.processing = False
+                    else:
+                        util.printInfo(1, self.username, "[!] 待唤醒！")
+                        if wsa_server.get_web_instance().is_connected(self.username):
+                            wsa_server.get_web_instance().add_cmd({"panelMsg": "[!] 待唤醒！", "Username" : self.username , 'robot': f'http://{cfg.fay_url}:5000/robot/Normal.jpg'})
+                        if wsa_server.get_instance().is_connected(self.username):
+                            content = {'Topic': 'Unreal', 'Data': {'Key': 'log', 'Value': "[!] 待唤醒！"}, 'Username' : self.username, 'robot': f'http://{cfg.fay_url}:5000/robot/Normal.jpg'}
+                            wsa_server.get_instance().add_cmd(content)
+
+            #非唤醒模式
+            else:
+                 self.on_speaking(text)
+                 self.processing = False
+        else:
+            #TODO 为什么这个设为False
+            # if self.wakeup_matched:
+            #     self.wakeup_matched = False
+            self.processing = False
+            util.printInfo(1, self.username, "[!] 语音未检测到内容！")
+            self.dynamic_threshold = self.__get_history_percentage(30)
+            if wsa_server.get_web_instance().is_connected(self.username):
+                wsa_server.get_web_instance().add_cmd({"panelMsg": "", 'Username' : self.username, 'robot': f'http://{cfg.fay_url}:5000/robot/Normal.jpg'})
+            if wsa_server.get_instance().is_connected(self.username):
+                content = {'Topic': 'Unreal', 'Data': {'Key': 'log', 'Value': ""}, 'Username' : self.username, 'robot': f'http://{cfg.fay_url}:5000/robot/Normal.jpg'}
+                wsa_server.get_instance().add_cmd(content)
+
+    def __record(self):   
+        try:
+            stream = self.get_stream() #通过此方法的阻塞来让程序往下执行
+        except Exception as e:
+                print(e)
+                util.printInfo(1, self.username, "请检查设备是否有误，再重新启动!")
+                return
+        isSpeaking = False
+        last_mute_time = time.time()
+        last_speaking_time = time.time()
+        data = None
+        concatenated_audio = bytearray()
+        audio_data_list = []
+        while self.__running:
+            try:
+                self.is_reading = True
+                data = stream.read(1024, exception_on_overflow=False)
+                self.is_reading = False
+            except Exception as e:
+                data = None
+                print(e)
+                util.log(1, "请检查录音设备是否有误，再重新启动!")
+                self.__running = False
+            if not data:
+                continue 
+
+            #是否可以拾音,不可以就掉弃录音
+            can_listen = True
+            #没有开唤醒，但面板或数字人正在播音时不能拾音
+            if cfg.config['source']['wake_word_enabled'] == False and self.__fay.speaking == True:
+                can_listen = False
+            
+            #普通唤醒模式已经激活，并且面板或数字人正在输出声音时不能拾音
+            if cfg.config['source']['wake_word_enabled'] == True and cfg.config['source']['wake_word_type'] == 'common' and self.wakeup_matched == True and self.__fay.speaking == True:
+                can_listen = False
+
+            if can_listen == False:#掉弃录音
+                data = None
+                continue
+            
+            #计算音量是否满足激活拾音
+            level = audioop.rms(data, 2)
+            if len(self.__history_data) >= 10:#保存激活前的音频，以免信息掉失
+                self.__history_data.pop(0)
+            if len(self.__history_level) >= 500:
+                self.__history_level.pop(0)
+            self.__history_data.append(data)
+            self.__history_level.append(level)
+            percentage = level / self.__MAX_LEVEL
+            history_percentage = self.__get_history_percentage(30)
+            if history_percentage > self.__dynamic_threshold:
+                self.__dynamic_threshold += (history_percentage - self.__dynamic_threshold) * 0.0025
+            elif history_percentage < self.__dynamic_threshold:
+                self.__dynamic_threshold += (history_percentage - self.__dynamic_threshold) * 1
+            
+            #激活拾音
+            if percentage > self.__dynamic_threshold:
+                last_speaking_time = time.time()
+                if not self.__processing and not isSpeaking and time.time() - last_mute_time > _ATTACK:
+                    isSpeaking = True  #用户正在说话
+                    util.printInfo(1, self.username,"聆听中...")
+                    if wsa_server.get_web_instance().is_connected(self.username):
+                        wsa_server.get_web_instance().add_cmd({"panelMsg": "聆听中...", 'Username' : self.username, 'robot': f'http://{cfg.fay_url}:5000/robot/Listening.jpg'})
+                    if wsa_server.get_instance().is_connected(self.username):
+                        content = {'Topic': 'Unreal', 'Data': {'Key': 'log', 'Value': "聆听中..."}, 'Username' : self.username, 'robot': f'http://{cfg.fay_url}:5000/robot/Listening.jpg'}
+                        wsa_server.get_instance().add_cmd(content)
+                    concatenated_audio.clear()
+                    self.__aLiNls = self.asrclient()
+                    try:
+                        self.__aLiNls.start()
+                    except Exception as e:
+                        print(e)
+                        util.printInfo(1, self.username, "aliyun asr 连接受限")
+                    for i in range(len(self.__history_data) - 1): #当前data在下面会做发送，这里是发送激活前的音频数据，以免漏掉信息
+                        buf = self.__history_data[i]
+                        audio_data_list.append(self.__process_audio_data(buf, self.channels))
+                        if self.ASRMode == "ali":
+                            self.__aLiNls.send(self.__process_audio_data(buf, self.channels).tobytes())
+                        else:
+                            concatenated_audio.extend(self.__process_audio_data(buf, self.channels).tobytes())
+                    self.__history_data.clear()
+            else:#结束拾音
+                last_mute_time = time.time()
+                if isSpeaking:
+                    if time.time() - last_speaking_time > _RELEASE: #TODO 更换的vad更靠谱
+                        isSpeaking = False
+                        self.__aLiNls.end()
+                        util.printInfo(1, self.username, "语音处理中...")
+                        self.__waitingResult(self.__aLiNls, concatenated_audio)
+
+                        mono_data = self.__concatenate_audio_data(audio_data_list)
+                        self.__save_audio_to_wav(mono_data, self.sample_rate, "cache_data/input.wav")
+                        audio_data_list = []
+            
+            #拾音中
+            if isSpeaking:
+                audio_data_list.append(self.__process_audio_data(data, self.channels))
+                if self.ASRMode == "ali":
+                    self.__aLiNls.send(self.__process_audio_data(data, self.channels).tobytes())
+                else:
+                    concatenated_audio.extend(self.__process_audio_data(data, self.channels).tobytes())
+        
+    def __save_audio_to_wav(self, data, sample_rate, filename):
+        # 确保数据类型为 int16
+        if data.dtype != np.int16:
+            data = data.astype(np.int16)
+        
+        # 打开 WAV 文件
+        with wave.open(filename, 'wb') as wf:
+            # 设置音频参数
+            n_channels = 1  # 单声道
+            sampwidth = 2   # 16 位音频，每个采样点 2 字节
+            wf.setnchannels(n_channels)
+            wf.setsampwidth(sampwidth)
+            wf.setframerate(sample_rate)
+            wf.writeframes(data.tobytes())
+
+    def __concatenate_audio_data(self, audio_data_list):
+        # 将累积的音频数据块连接起来
+        data = np.concatenate(audio_data_list)
+        return data
+    
+    #转变为单声道np.int16
+    def __process_audio_data(self, data, channels):
+        data = bytearray(data)
+        # 将字节数据转换为 numpy 数组
+        data = np.frombuffer(data, dtype=np.int16)
+        # 重塑数组，将数据分离成多个声道
+        data = np.reshape(data, (-1, channels))
+        # 对所有声道的数据进行平均，生成单声道
+        mono_data = np.mean(data, axis=1).astype(np.int16)
+        return mono_data
+     
+    def set_processing(self, processing):
+        self.__processing = processing
+
+    def start(self):
+        MyThread(target=self.__record).start()
+
+    def stop(self):
+        self.__running = False
+
+    @abstractmethod
+    def on_speaking(self, text):
+        pass
+
+    #TODO Edit by xszyou on 20230113:把流的获取方式封装出来方便实现麦克风录制及网络流等不同的流录制子类
+    @abstractmethod
+    def get_stream(self):
+        pass
+
+    @abstractmethod
+    def is_remote(self):
+        pass

core/wsa_server.py

@@ -0,0 +1,289 @@
+from asyncio import AbstractEventLoop
+
+import websockets
+import asyncio
+import json
+from abc import abstractmethod
+from websockets.legacy.server import Serve
+
+from utils import util
+from scheduler.thread_manager import MyThread
+
+class MyServer:
+    def __init__(self, host='0.0.0.0', port=10000):
+        self.lock = asyncio.Lock()
+        self.__host = host  # ip
+        self.__port = port  # 端口号
+        self.__listCmd = []  # 要发送的信息的列表
+        self.__clients = list() 
+        self.__server: Serve = None
+        self.__event_loop: AbstractEventLoop = None
+        self.__running = True
+        self.__pending = None
+        self.isConnect = False
+        self.TIMEOUT = 3  # 设置任何超时时间为 3 秒
+        self.__tasks = {}  # 记录任务和开始时间的字典
+
+    # 接收处理
+    async def __consumer_handler(self, websocket, path):
+        username = None
+        try:
+            async for message in websocket:
+                await asyncio.sleep(0.01)
+                try:
+                    username = json.loads(message).get("Username")
+                except json.JSONDecodeError as e:#json格式有误，不处理
+                    pass
+                if username:
+                    remote_address = websocket.remote_address
+                    unique_id = f"{remote_address[0]}:{remote_address[1]}"
+                    async with self.lock:
+                        for i in range(len(self.__clients)): 
+                            if self.__clients[i]["id"] == unique_id:
+                                self.__clients[i]["username"] = username
+                await self.__consumer(message)
+        except websockets.exceptions.ConnectionClosedError as e:
+            # 从客户端列表中移除已断开的连接
+            await self.remove_client(websocket)
+            util.printInfo(1, "User" if  username is None else username, f"WebSocket 连接关闭: {e}")
+
+    # 发送处理        
+    async def __producer_handler(self, websocket, path):
+        while self.__running:
+            await asyncio.sleep(0.01)
+            if len(self.__listCmd) > 0:
+                message = await self.__producer()
+                if message:
+                    username = json.loads(message).get("Username")
+                    if username is None:
+                        # 群发消息
+                        async with self.lock:
+                            wsclients = [c["websocket"] for c in self.__clients]
+                        tasks = [self.send_message_with_timeout(client, message, username, timeout=3) for client in wsclients]
+                        await asyncio.gather(*tasks)
+                    else:
+                        # 向指定用户发送消息
+                        async with self.lock:
+                            target_clients = [c["websocket"] for c in self.__clients if c.get("username") == username]
+                        tasks = [self.send_message_with_timeout(client, message, username, timeout=3) for client in target_clients]
+                        await asyncio.gather(*tasks)
+
+    # 发送消息（设置超时）
+    async def send_message_with_timeout(self, client, message, username, timeout=3):
+        try:
+            await asyncio.wait_for(self.send_message(client, message, username), timeout=timeout)
+        except asyncio.TimeoutError:
+            util.printInfo(1, "User" if username is None else username, f"发送消息超时: 用户名 {username}")
+        except websockets.exceptions.ConnectionClosed as e:
+            # 从客户端列表中移除已断开的连接
+            await self.remove_client(client)
+            util.printInfo(1, "User" if username is None else username, f"WebSocket 连接关闭: {e}")
+
+    # 发送消息
+    async def send_message(self, client, message, username):
+        try:
+            await client.send(message)
+        except websockets.exceptions.ConnectionClosed as e:
+            # 从客户端列表中移除已断开的连接
+            await self.remove_client(client)
+            util.printInfo(1, "User" if username is None else username, f"WebSocket 连接关闭: {e}")
+
+                
+    async def __handler(self, websocket, path):
+        self.isConnect = True
+        util.log(1,"websocket连接上:{}".format(self.__port))
+        self.on_connect_handler()
+        remote_address = websocket.remote_address
+        unique_id = f"{remote_address[0]}:{remote_address[1]}"
+        async with self.lock:
+            self.__clients.append({"id" : unique_id, "websocket" : websocket, "username" : "User"})
+        consumer_task = asyncio.create_task(self.__consumer_handler(websocket, path))#接收
+        producer_task = asyncio.create_task(self.__producer_handler(websocket, path))#发送
+        done, self.__pending = await asyncio.wait([consumer_task, producer_task], return_when=asyncio.FIRST_COMPLETED)
+
+        for task in self.__pending:
+            task.cancel()
+            try:
+                await task
+            except asyncio.CancelledError:
+                pass
+            
+        # 从客户端列表中移除已断开的连接
+        await self.remove_client(websocket)
+        util.log(1, "websocket连接断开:{}".format(unique_id))
+                
+    async def __consumer(self, message):
+        self.on_revice_handler(message)
+    
+    async def __producer(self):
+        if len(self.__listCmd) > 0:
+            message = self.on_send_handler(self.__listCmd.pop(0))
+            return message
+        else:
+            return None
+        
+    async def remove_client(self, websocket):
+        async with self.lock:
+            self.__clients = [c for c in self.__clients if c["websocket"] != websocket]
+            if len(self.__clients) == 0:
+                self.isConnect = False
+        self.on_close_handler()
+
+    def is_connected(self, username):
+        if username is None:
+            username = "User"
+        if len(self.__clients) == 0:
+            return False
+        clients = [c for c in self.__clients if c["username"] == username]
+        if len(clients) > 0:
+            return True
+        return False
+
+
+    #Edit by xszyou on 20230113:通过继承此类来实现服务端的接收后处理逻辑
+    @abstractmethod
+    def on_revice_handler(self, message):
+        pass
+
+    #Edit by xszyou on 20230114:通过继承此类来实现服务端的连接处理逻辑
+    @abstractmethod
+    def on_connect_handler(self):
+        pass
+    
+    #Edit by xszyou on 20230804:通过继承此类来实现服务端的发送前的处理逻辑
+    @abstractmethod
+    def on_send_handler(self, message):
+        return message
+
+    #Edit by xszyou on 20230816:通过继承此类来实现服务端的断开后的处理逻辑
+    @abstractmethod
+    def on_close_handler(self):
+        pass
+
+    # 创建server
+    def __connect(self):
+        self.__event_loop = asyncio.new_event_loop()
+        asyncio.set_event_loop(self.__event_loop)
+        self.__isExecute = True
+        if self.__server:
+            util.log(1, 'server already exist')
+            return
+        self.__server = websockets.serve(self.__handler, self.__host, self.__port)
+        asyncio.get_event_loop().run_until_complete(self.__server)
+        asyncio.get_event_loop().run_forever()
+
+    # 往要发送的命令列表中，添加命令
+    def add_cmd(self, content):
+        if not self.__running:
+            return
+        jsonStr = json.dumps(content)
+        self.__listCmd.append(jsonStr)
+        # util.log('命令 {}'.format(content))
+
+    # 开启服务
+    def start_server(self):
+        MyThread(target=self.__connect).start()
+
+    # 关闭服务
+    def stop_server(self):
+        self.__running = False
+        self.isConnect = False
+        if self.__server is None:
+            return
+        self.__server.close()
+        self.__server = None
+        self.__clients = []
+        util.log(1, "WebSocket server stopped.")
+
+
+#ui端server
+class WebServer(MyServer):
+    def __init__(self, host='0.0.0.0', port=10003):
+        super().__init__(host, port)
+
+    def on_revice_handler(self, message):
+        pass
+    
+    def on_connect_handler(self):
+        self.add_cmd({"panelMsg": "使用提示：Fay可以独立使用，启动数字人将自动对接。"})
+
+    def on_send_handler(self, message):
+        return message
+
+    def on_close_handler(self):
+        pass
+
+#数字人端server
+class HumanServer(MyServer):
+    def __init__(self, host='0.0.0.0', port=10002):
+        super().__init__(host, port)
+
+    def on_revice_handler(self, message):
+       pass
+
+    def on_connect_handler(self):
+        web_server_instance = get_web_instance()  
+        web_server_instance.add_cmd({"is_connect": self.isConnect}) 
+        
+
+    def on_send_handler(self, message):
+        # util.log(1, '向human发送 {}'.format(message))
+        if not self.isConnect:
+            return None
+        return message
+
+    def on_close_handler(self):
+        web_server_instance = get_web_instance()  
+        web_server_instance.add_cmd({"is_connect": self.isConnect}) 
+
+        
+
+#测试
+class TestServer(MyServer):
+    def __init__(self, host='0.0.0.0', port=10000):
+        super().__init__(host, port)
+
+    def on_revice_handler(self, message):
+        print(message)
+    
+    def on_connect_handler(self):
+        print("连接上了")
+    
+    def on_send_handler(self, message):
+        return message
+    
+    def on_close_handler(self):
+        pass
+
+
+
+#单例
+
+__instance: MyServer = None
+__web_instance: MyServer = None
+
+
+def new_instance(host='0.0.0.0', port=10002) -> MyServer:
+    global __instance
+    if __instance is None:
+        __instance = HumanServer(host, port)
+    return __instance
+
+
+def new_web_instance(host='0.0.0.0', port=10003) -> MyServer:
+    global __web_instance
+    if __web_instance is None:
+        __web_instance = WebServer(host, port)
+    return __web_instance
+
+
+def get_instance() -> MyServer:
+    return __instance
+
+
+def get_web_instance() -> MyServer:
+    return __web_instance
+
+if __name__ == '__main__':
+    testServer = TestServer(host='0.0.0.0', port=10000)
+    testServer.start_server()

docker/Dockerfile

@@ -0,0 +1,10 @@
+FROM docker.m.daocloud.io/python:3.10
+
+COPY install_deps.sh /usr/local/bin/install_deps.sh
+RUN chmod +x /usr/local/bin/install_deps.sh && /usr/local/bin/install_deps.sh
+RUN pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple/
+COPY requirements.txt /app/
+WORKDIR /app
+RUN pip install --no-cache-dir -r requirements.txt
+COPY ./ /app
+CMD ["python", "main.py"]

docker/environment.yml

@@ -0,0 +1,32 @@
+name: fay
+channels:
+  - defaults
+dependencies:
+  - python=3.10
+  - requests
+  - numpy
+  - pyaudio=0.2.11
+  - websockets=10.2
+  - ws4py=0.5.1
+  - pyqt=5.15.6
+  - flask=3.0.0
+  - openpyxl=3.0.9
+  - flask-cors=3.0.10
+  - pyqtwebengine=5.15.5
+  - eyed3=0.9.6
+  - websocket-client
+  - azure-cognitiveservices-speech
+  - aliyun-python-sdk-core
+  - simhash
+  - pytz
+  - gevent=22.10.1
+  - edge-tts=6.1.3
+  - ultralytics=8.0.2
+  - pydub
+  - cemotion
+  - langchain=0.0.336
+  - chromadb
+  - tenacity=8.2.3
+  - pygame
+  - scipy
+  - pip

docker/install_deps.sh

@@ -0,0 +1,50 @@
+#!/bin/bash
+
+# 检测 Debian 系统（如 Ubuntu）
+if grep -qEi "(debian|ubuntu)" /etc/*release; then
+    apt-get update -yq --fix-missing && \
+    DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends \
+    pkg-config \
+    wget \
+    cmake \
+    curl \
+    git \
+    vim \
+    build-essential \
+    libgl1-mesa-glx \
+    portaudio19-dev \
+    libnss3 \
+    libxcomposite1 \
+    libxrender1 \
+    libxrandr2 \
+    libqt5webkit5-dev \
+    libxdamage1 \
+    libxtst6 && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+# 检测 CentOS 系统
+elif grep -qEi "(centos|fedora|rhel)" /etc/*release; then
+    yum update -y && \
+    yum install -y \
+    pkgconfig \
+    wget \
+    cmake \
+    curl \
+    git \
+    vim-enhanced \
+    gcc \
+    gcc-c++ \
+    mesa-libGL \
+    portaudio \
+    nss \
+    libXcomposite \
+    libXrender \
+    libXrandr \
+    qt5-qtwebkit-devel \
+    libXdamage \
+    libXtst && \
+    yum clean all
+else
+    echo "Unsupported OS"
+    exit 1
+fi

docker/requirements.txt

@@ -0,0 +1,28 @@
+requests  
+numpy
+pyaudio~=0.2.11
+websockets~=10.2
+ws4py~=0.5.1
+pyqt5~=5.15.6
+flask~=3.0.0
+openpyxl~=3.0.9
+flask_cors~=3.0.10
+PyQtWebEngine~=5.15.5
+eyed3~=0.9.6
+websocket-client
+azure-cognitiveservices-speech
+aliyun-python-sdk-core
+simhash
+pytz
+gevent~=22.10.1
+edge_tts~=6.1.3
+eyed3
+ultralytics~=8.0.2
+pydub
+cemotion
+langchain==0.0.336
+chromadb
+tenacity==8.2.3
+pygame
+scipy
+flask-httpauth

fay_booter.py

@@ -0,0 +1,393 @@
+#核心启动模块
+import time
+import re
+import pyaudio
+import socket
+import psutil
+import sys
+import requests
+from core.interact import Interact
+from core.recorder import Recorder
+from core import fay_core
+from scheduler.thread_manager import MyThread
+from utils import util, config_util, stream_util
+from core.wsa_server import MyServer
+from scheduler.thread_manager import MyThread
+from core import wsa_server
+
+feiFei: fay_core.FeiFei = None
+recorderListener: Recorder = None
+__running = False
+deviceSocketServer = None
+DeviceInputListenerDict = {}
+ngrok = None
+
+#启动状态
+def is_running():
+    return __running
+
+#录制麦克风音频输入并传给aliyun
+class RecorderListener(Recorder):
+
+    def __init__(self, device, fei):
+        self.__device = device
+        self.__RATE = 16000
+        self.__FORMAT = pyaudio.paInt16
+        self.__running = False
+        self.username = 'User'
+        self.channels = 1
+        self.sample_rate = 16000
+
+        super().__init__(fei)
+
+    def on_speaking(self, text):
+        if len(text) > 1:
+            interact = Interact("mic", 1, {'user': 'User', 'msg': text})
+            util.printInfo(3, "语音", '{}'.format(interact.data["msg"]), time.time())
+            feiFei.on_interact(interact)
+
+    def get_stream(self):
+        try:
+            self.paudio = pyaudio.PyAudio()
+            device_id = 0  # 或者根据需要选择其他设备
+
+            # 获取设备信息
+            device_info = self.paudio.get_device_info_by_index(device_id)
+            self.channels = device_info.get('maxInputChannels', 1) #很多麦克风只支持单声道录音
+            # self.sample_rate = int(device_info.get('defaultSampleRate', self.__RATE))
+            print(self.sample_rate)
+
+            # 设置格式（这里以16位深度为例）
+            format = pyaudio.paInt16
+
+            # 打开音频流，使用设备的最大声道数和默认采样率
+            self.stream = self.paudio.open(
+                input_device_index=device_id,
+                rate=self.sample_rate,
+                format=format,
+                channels=self.channels,
+                input=True,
+                frames_per_buffer=4096
+            )
+
+            self.__running = True
+            MyThread(target=self.__pyaudio_clear).start()
+        except Exception as e:
+            print(f"Error: {e}")
+            time.sleep(10)
+        return self.stream
+
+
+    def __pyaudio_clear(self):
+        while self.__running:
+            time.sleep(30)
+    
+    def stop(self):
+        super().stop()
+        self.__running = False
+        try:
+            while self.is_reading:
+                time.sleep(0.1)
+            self.stream.stop_stream()
+            self.stream.close()
+            self.paudio.terminate()
+        except Exception as e:
+                print(e)
+                util.log(1, "请检查设备是否有误，再重新启动!")
+
+    def is_remote(self):
+        return False
+                    
+        
+
+
+#Edit by xszyou on 20230113:录制远程设备音频输入并传给aliyun
+class DeviceInputListener(Recorder):
+    def __init__(self, deviceConnector, fei):
+        super().__init__(fei)
+        self.__running = True
+        self.streamCache = None
+        self.thread = MyThread(target=self.run)
+        self.thread.start()  #启动远程音频输入设备监听线程
+        self.username = 'User'
+        self.isOutput = True
+        self.deviceConnector = deviceConnector
+
+    def run(self):
+        #启动ngork
+        self.streamCache = stream_util.StreamCache(1024*1024*20)
+        addr = None
+        while self.__running:
+            try:
+                
+                data = b""
+                while self.deviceConnector:
+                    data = self.deviceConnector.recv(2048)
+                    if b"<username>" in data:
+                        data_str = data.decode("utf-8")
+                        match = re.search(r"<username>(.*?)</username>", data_str)
+                        if match:
+                            self.username = match.group(1)
+                        else:
+                            self.streamCache.write(data)
+                    if b"<output>" in data:
+                        data_str = data.decode("utf-8")
+                        match = re.search(r"<output>(.*?)<output>", data_str)
+                        if match:
+                            self.isOutput = (match.group(1) == "True")
+                        else:
+                            self.streamCache.write(data)
+                    if not b"<username>" in data and not b"<output>" in data:
+                        self.streamCache.write(data)
+                    time.sleep(0.005)
+                self.streamCache.clear()
+         
+            except Exception as err:
+                pass
+            time.sleep(1)
+
+    def on_speaking(self, text):
+        global feiFei
+        if len(text) > 1:
+            interact = Interact("socket", 1, {"user": self.username, "msg": text, "socket": self.deviceConnector})
+            util.printInfo(3, "(" + self.username + ")远程音频输入", '{}'.format(interact.data["msg"]), time.time())
+            feiFei.on_interact(interact)
+
+    #recorder会等待stream不为空才开始录音
+    def get_stream(self):
+        while not self.deviceConnector:
+            time.sleep(1)
+            pass
+        return self.streamCache
+
+    def stop(self):
+        super().stop()
+        self.__running = False
+
+    def is_remote(self):
+        return True
+
+#检查远程音频连接状态
+def device_socket_keep_alive():
+    global DeviceInputListenerDict
+    while __running:
+        delkey = None
+        for key, value in DeviceInputListenerDict.items():
+            try:
+                value.deviceConnector.send(b'\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8')#发送心跳包
+                if wsa_server.get_web_instance().is_connected(value.username):
+                    wsa_server.get_web_instance().add_cmd({"remote_audio_connect": True, "Username" : value.username}) 
+            except Exception as serr:
+                util.printInfo(3, value.username, "远程音频输入输出设备已经断开：{}".format(key))
+                value.stop()
+                delkey = key
+                break
+        if delkey:
+             value =  DeviceInputListenerDict.pop(delkey)
+             if wsa_server.get_web_instance().is_connected(value.username):
+                wsa_server.get_web_instance().add_cmd({"remote_audio_connect": False, "Username" : value.username})
+        time.sleep(1)
+
+#远程音频连接
+def accept_audio_device_output_connect():
+    global deviceSocketServer
+    global __running
+    global DeviceInputListenerDict
+    deviceSocketServer = socket.socket(socket.AF_INET,socket.SOCK_STREAM) 
+    deviceSocketServer.bind(("0.0.0.0",10001))   
+    deviceSocketServer.listen(1)
+    MyThread(target = device_socket_keep_alive).start() # 开启心跳包检测
+    addr = None        
+    
+    while __running:
+        try:
+            deviceConnector,addr = deviceSocketServer.accept()   #接受TCP连接，并返回新的套接字与IP地址
+            deviceInputListener = DeviceInputListener(deviceConnector, feiFei)  # 设备音频输入输出麦克风
+            deviceInputListener.start()
+
+            #把DeviceInputListenner对象记录下来
+            peername = str(deviceConnector.getpeername()[0]) + ":" + str(deviceConnector.getpeername()[1])
+            DeviceInputListenerDict[peername] = deviceInputListener
+            util.log(1,"远程音频输入输出设备连接上：{}".format(addr))
+        except Exception as e:
+            pass
+
+def kill_process_by_port(port):
+    for proc in psutil.process_iter(['pid', 'name','cmdline']):
+        try:
+            for conn in proc.connections(kind='inet'):
+                if conn.laddr.port == port:
+                    proc.terminate()
+                    proc.wait()
+        except(psutil.NosuchProcess, psutil.AccessDenied):
+            pass
+#数字人端请求获取最新的自动播放消息，若自动播放服务关闭会自动退出自动播放
+def start_auto_play_service():
+    url = f"{config_util.config['source']['automatic_player_url']}/get_auto_play_item"
+    user = "User" #TODO 临时固死了
+    is_auto_server_error = False
+    while __running:
+        if config_util.config['source']['wake_word_enabled'] and config_util.config['source']['wake_word_type'] == 'common' and recorderListener.wakeup_matched == True:
+            time.sleep(0.01)
+            continue
+        if is_auto_server_error:
+            util.printInfo(1, user, '60s后重连自动播放服务器')
+            time.sleep(60)
+        # 请求自动播放服务器
+        with fay_core.auto_play_lock:
+            if config_util.config['source']['automatic_player_status'] and config_util.config['source']['automatic_player_url'] is not None and fay_core.can_auto_play == True and (config_util.config["interact"]["playSound"] or wsa_server.get_instance().is_connected(user)):
+                fay_core.can_auto_play = False
+                post_data = {"user": user}
+                try:
+                    response = requests.post(url, json=post_data, timeout=5)
+                    if response.status_code == 200:
+                        is_auto_server_error = False
+                        data = response.json()
+                        audio_url = data.get('audio')
+                        if not audio_url or audio_url.strip()[0:4] != "http":
+                            audio_url = None   
+                        response_text = data.get('text')
+                        timestamp = data.get('timestamp')
+                        interact = Interact("auto_play", 2, {'user': user, 'text': response_text, 'audio': audio_url})
+                        util.printInfo(1, user, '自动播放：{}，{}'.format(response_text, audio_url), time.time())
+                        feiFei.on_interact(interact)
+                    else:
+                        is_auto_server_error = True
+                        fay_core.can_auto_play = True
+                        util.printInfo(1, user, '请求自动播放服务器失败，错误代码是：{}'.format(response.status_code))
+                except requests.exceptions.RequestException as e:
+                    is_auto_server_error = True
+                    fay_core.can_auto_play = True
+                    util.printInfo(1, user, '请求自动播放服务器失败，错误信息是：{}'.format(e))
+        time.sleep(0.01)
+     
+#控制台输入监听
+def console_listener():
+    global feiFei
+    while __running:
+        try:
+            text = input()
+        except EOFError:
+            util.log(1, "控制台已经关闭")
+            break
+        
+        args = text.split(' ')
+
+        if len(args) == 0 or len(args[0]) == 0:
+            continue
+
+        if args[0] == 'help':
+            util.log(1, 'in <msg> \t通过控制台交互')
+            util.log(1, 'restart \t重启服务')
+            util.log(1, 'stop \t\t关闭服务')
+            util.log(1, 'exit \t\t结束程序')
+
+        elif args[0] == 'stop':
+            stop()
+            break
+
+        elif args[0] == 'restart':
+            stop()
+            time.sleep(0.1)
+            start()
+
+        elif args[0] == 'in':
+            if len(args) == 1:
+                util.log(1, '错误的参数！')
+            msg = text[3:len(text)]
+            util.printInfo(3, "控制台", '{}: {}'.format('控制台', msg))
+            interact = Interact("console", 1, {'user': 'User', 'msg': msg})
+            thr = MyThread(target=feiFei.on_interact, args=[interact])
+            thr.start()
+
+        elif args[0]=='exit':
+            stop()
+            time.sleep(0.1)
+            util.log(1,'程序正在退出..')
+            ports =[10001,10002,10003,5000]
+            for port in ports:
+                kill_process_by_port(port)
+            sys.exit(0)
+        else:
+            util.log(1, '未知命令！使用 \'help\' 获取帮助.')
+
+#停止服务
+def stop():
+    global feiFei
+    global recorderListener
+    global __running
+    global DeviceInputListenerDict
+    global ngrok
+
+    util.log(1, '正在关闭服务...')
+    __running = False
+    if recorderListener is not None:
+        util.log(1, '正在关闭录音服务...')
+        recorderListener.stop()
+        time.sleep(0.1)
+    util.log(1, '正在关闭远程音频输入输出服务...')
+    if len(DeviceInputListenerDict) > 0:
+        for key in list(DeviceInputListenerDict.keys()):
+            value = DeviceInputListenerDict.pop(key)
+            value.stop()
+    deviceSocketServer.close()
+    util.log(1, '正在关闭核心服务...')
+    feiFei.stop()
+    util.log(1, '服务已关闭！')
+
+
+#开启服务
+def start():
+    global feiFei
+    global recorderListener
+    global __running
+    util.log(1, '开启服务...')
+    __running = True
+
+    #读取配置
+    util.log(1, '读取配置...')
+    config_util.load_config()
+
+    #开启核心服务
+    util.log(1, '开启核心服务...')
+    feiFei = fay_core.FeiFei()
+    feiFei.start()
+
+    #加载本地知识库
+    if config_util.key_chat_module == 'langchain':
+        from llm import nlp_langchain
+        nlp_langchain.save_all()
+    if config_util.key_chat_module == 'privategpt':    
+        from llm import nlp_privategpt
+        nlp_privategpt.save_all()
+
+    #开启录音服务
+    record = config_util.config['source']['record']
+    if record['enabled']:
+        util.log(1, '开启录音服务...')
+        recorderListener = RecorderListener(record['device'], feiFei)  # 监听麦克风
+        recorderListener.start()
+
+    #启动声音沟通接口服务
+    util.log(1,'启动声音沟通接口服务...')
+    deviceSocketThread = MyThread(target=accept_audio_device_output_connect)
+    deviceSocketThread.start()
+
+    #启动自动播放服务
+    util.log(1,'启动自动播放服务...')
+    MyThread(target=start_auto_play_service).start()
+            
+    #监听控制台
+    util.log(1, '注册命令...')
+    MyThread(target=console_listener).start()  # 监听控制台
+
+    util.log(1, '服务启动完成!')
+    util.log(1, '使用 \'help\' 获取帮助.')
+
+    
+
+if __name__ == '__main__':
+    ws_server: MyServer = None
+    feiFei: fay_core.FeiFei = None
+    recorderListener: Recorder = None
+    start()

gui/flask_server.py

@@ -0,0 +1,359 @@
+import importlib
+import json
+import time
+import os
+import pyaudio
+import re
+from flask import Flask, render_template, request, jsonify, Response, send_file
+from flask_cors import CORS
+import requests
+
+import fay_booter
+
+from tts import tts_voice
+from gevent import pywsgi
+from scheduler.thread_manager import MyThread
+from utils import config_util, util
+from core import wsa_server
+from core import fay_core
+from core import content_db
+from core.interact import Interact
+from core import member_db
+import fay_booter
+from flask_httpauth import HTTPBasicAuth
+
+
+__app = Flask(__name__)
+auth = HTTPBasicAuth()
+CORS(__app, supports_credentials=True)
+
+def load_users():
+    with open('verifier.json') as f:
+        users = json.load(f)
+    return users
+
+users = load_users()
+
+@auth.verify_password
+def verify_password(username, password):
+    if not users or config_util.start_mode == 'common':
+        return True
+    if username in users and users[username] == password:
+        return username
+
+
+def __get_template():
+    return render_template('index.html')
+
+
+def __get_device_list():
+    if config_util.start_mode == 'common':
+        audio = pyaudio.PyAudio()
+        device_list = []
+        for i in range(audio.get_device_count()):
+            devInfo = audio.get_device_info_by_index(i)
+            if devInfo['hostApi'] == 0:
+                device_list.append(devInfo["name"])
+        
+        return list(set(device_list))
+    else:
+        return []
+
+
+@__app.route('/api/submit', methods=['post'])
+def api_submit():
+    data = request.values.get('data')
+    config_data = json.loads(data)
+    if(config_data['config']['source']['record']['enabled']):
+        config_data['config']['source']['record']['channels'] = 0
+        audio = pyaudio.PyAudio()
+        for i in range(audio.get_device_count()):
+            devInfo = audio.get_device_info_by_index(i)
+            if devInfo['name'].find(config_data['config']['source']['record']['device']) >= 0 and devInfo['hostApi'] == 0:
+                 config_data['config']['source']['record']['channels'] = devInfo['maxInputChannels']
+
+    config_util.save_config(config_data['config'])
+
+    return '{"result":"successful"}'
+
+@__app.route('/api/get-data', methods=['post'])
+def api_get_data():
+    config_util.load_config()
+    voice_list = tts_voice.get_voice_list()
+    send_voice_list = []
+    if config_util.tts_module == 'ali':
+        wsa_server.get_web_instance().add_cmd({
+        "voiceList": [
+            {"id": "abin", "name": "阿斌"},
+            {"id": "zhixiaobai", "name": "知小白"},
+            {"id": "zhixiaoxia", "name": "知小夏"},
+            {"id": "zhixiaomei", "name": "知小妹"},
+            {"id": "zhigui", "name": "知柜"},
+            {"id": "zhishuo", "name": "知硕"},
+            {"id": "aixia", "name": "艾夏"},
+            {"id": "zhifeng_emo", "name": "知锋_多情感"},
+            {"id": "zhibing_emo", "name": "知冰_多情感"},
+            {"id": "zhimiao_emo", "name": "知妙_多情感"},
+            {"id": "zhimi_emo", "name": "知米_多情感"},
+            {"id": "zhiyan_emo", "name": "知燕_多情感"},
+            {"id": "zhibei_emo", "name": "知贝_多情感"},
+            {"id": "zhitian_emo", "name": "知甜_多情感"},
+            {"id": "xiaoyun", "name": "小云"},
+            {"id": "xiaogang", "name": "小刚"},
+            {"id": "ruoxi", "name": "若兮"},
+            {"id": "siqi", "name": "思琪"},
+            {"id": "sijia", "name": "思佳"},
+            {"id": "sicheng", "name": "思诚"},
+            {"id": "aiqi", "name": "艾琪"},
+            {"id": "aijia", "name": "艾佳"},
+            {"id": "aicheng", "name": "艾诚"},
+            {"id": "aida", "name": "艾达"},
+            {"id": "ninger", "name": "宁儿"},
+            {"id": "ruilin", "name": "瑞琳"},
+            {"id": "siyue", "name": "思悦"},
+            {"id": "aiya", "name": "艾雅"},
+            {"id": "aimei", "name": "艾美"},
+            {"id": "aiyu", "name": "艾雨"},
+            {"id": "aiyue", "name": "艾悦"},
+            {"id": "aijing", "name": "艾婧"},
+            {"id": "xiaomei", "name": "小美"},
+            {"id": "aina", "name": "艾娜"},
+            {"id": "yina", "name": "伊娜"},
+            {"id": "sijing", "name": "思婧"},
+            {"id": "sitong", "name": "思彤"},
+            {"id": "xiaobei", "name": "小北"},
+            {"id": "aitong", "name": "艾彤"},
+            {"id": "aiwei", "name": "艾薇"},
+            {"id": "aibao", "name": "艾宝"},
+            {"id": "shanshan", "name": "姗姗"},
+            {"id": "chuangirl", "name": "小玥"},
+            {"id": "lydia", "name": "Lydia"},
+            {"id": "aishuo", "name": "艾硕"},
+            {"id": "qingqing", "name": "青青"},
+            {"id": "cuijie", "name": "翠姐"},
+            {"id": "xiaoze", "name": "小泽"},
+            {"id": "zhimao", "name": "知猫"},
+            {"id": "zhiyuan", "name": "知媛"},
+            {"id": "zhiya", "name": "知雅"},
+            {"id": "zhiyue", "name": "知悦"},
+            {"id": "zhida", "name": "知达"},
+            {"id": "zhistella", "name": "知莎"},
+            {"id": "kelly", "name": "Kelly"},
+            {"id": "jiajia", "name": "佳佳"},
+            {"id": "taozi", "name": "桃子"},
+            {"id": "guijie", "name": "柜姐"},
+            {"id": "stella", "name": "Stella"},
+            {"id": "stanley", "name": "Stanley"},
+            {"id": "kenny", "name": "Kenny"},
+            {"id": "rosa", "name": "Rosa"},
+            {"id": "mashu", "name": "马树"},
+            {"id": "xiaoxian", "name": "小仙"},
+            {"id": "yuer", "name": "悦儿"},
+            {"id": "maoxiaomei", "name": "猫小美"},
+            {"id": "aifei", "name": "艾飞"},
+            {"id": "yaqun", "name": "亚群"},
+            {"id": "qiaowei", "name": "巧薇"},
+            {"id": "dahu", "name": "大虎"},
+            {"id": "ailun", "name": "艾伦"},
+            {"id": "jielidou", "name": "杰力豆"},
+            {"id": "laotie", "name": "老铁"},
+            {"id": "laomei", "name": "老妹"},
+            {"id": "aikan", "name": "艾侃"}
+        ]
+    })
+    elif config_util.tts_module == 'volcano':
+        wsa_server.get_web_instance().add_cmd({
+        "voiceList": [
+            {"id": "BV001_streaming", "name": "通用女声"},
+            {"id": "BV002_streaming", "name": "通用男声"},
+            {"id": "zh_male_jingqiangkanye_moon_bigtts", "name": "京腔侃爷/Harmony"},
+            {"id": "zh_female_shuangkuaisisi_moon_bigtts", "name": "爽快思思/Skye"},
+            {"id": "zh_male_wennuanahu_moon_bigtts", "name": "温暖阿虎/Alvin"},
+            {"id": "zh_female_wanwanxiaohe_moon_bigtts", "name": "湾湾小何"},
+        ]
+    })    
+    else:
+        voice_list = tts_voice.get_voice_list()
+        send_voice_list = []
+        for voice in voice_list: 
+            voice_data = voice.value 
+            send_voice_list.append({"id": voice_data['name'], "name": voice_data['name']})
+        wsa_server.get_web_instance().add_cmd({
+            "voiceList": send_voice_list
+        })
+    wsa_server.get_web_instance().add_cmd({"deviceList": __get_device_list()})
+    if fay_booter.is_running():
+        wsa_server.get_web_instance().add_cmd({"liveState": 1})
+    return json.dumps({'config': config_util.config, 'voice_list' : send_voice_list})
+
+
+@__app.route('/api/start-live', methods=['post'])
+def api_start_live():
+    # time.sleep(5)
+    fay_booter.start()
+    time.sleep(1)
+    wsa_server.get_web_instance().add_cmd({"liveState": 1})
+    return '{"result":"successful"}'
+
+
+@__app.route('/api/stop-live', methods=['post'])
+def api_stop_live():
+    # time.sleep(1)
+    fay_booter.stop()
+    time.sleep(1)
+    wsa_server.get_web_instance().add_cmd({"liveState": 0})
+    return '{"result":"successful"}'
+
+@__app.route('/api/send', methods=['post'])
+def api_send():
+    data = request.values.get('data')
+    info = json.loads(data)
+    interact = Interact("text", 1, {'user': info['username'], 'msg': info['msg']})
+    util.printInfo(3, "文字发送按钮", '{}'.format(interact.data["msg"]), time.time())
+    fay_booter.feiFei.on_interact(interact)
+    return '{"result":"successful"}'
+
+#获取指定用户的消息记录
+@__app.route('/api/get-msg', methods=['post'])
+def api_get_Msg():
+    data = request.form.get('data')
+    data = json.loads(data)
+    uid = member_db.new_instance().find_user(data["username"])
+    contentdb = content_db.new_instance()
+    if uid == 0:
+        return json.dumps({'list': []})
+    else:
+        list = contentdb.get_list('all','desc',1000, uid)
+    relist = []
+    i = len(list)-1
+    while i >= 0:
+        relist.append(dict(type=list[i][0], way=list[i][1], content=list[i][2], createtime=list[i][3], timetext=list[i][4], username=list[i][5]))
+        i -= 1
+    if fay_booter.is_running():
+        wsa_server.get_web_instance().add_cmd({"liveState": 1})
+    return json.dumps({'list': relist})
+
+@__app.route('/v1/chat/completions', methods=['post'])
+@__app.route('/api/send/v1/chat/completions', methods=['post'])
+def api_send_v1_chat_completions():
+    data = request.json  
+    last_content = ""
+    if 'messages' in data and data['messages']:
+        last_message = data['messages'][-1]  
+        username = last_message.get('role', 'User')  
+        if username == 'user':
+            username = 'User'
+        last_content = last_message.get('content', 'No content provided')  
+    else:
+        last_content = 'No messages found'
+        username = 'User'
+
+    model = data.get('model', 'fay')
+    interact = Interact("text", 1, {'user': username, 'msg': last_content})
+    util.printInfo(3, "文字沟通接口", '{}'.format(interact.data["msg"]), time.time())
+    text = fay_booter.feiFei.on_interact(interact)
+
+    if model == 'fay-streaming':
+        return stream_response(text)
+    else:
+        return non_streaming_response(last_content, text)
+
+@__app.route('/api/get-member-list', methods=['post'])
+def api_get_Member_list():
+    memberdb = member_db.new_instance()
+    list = memberdb.get_all_users()
+    return json.dumps({'list': list})
+
+
+def stream_response(text):
+    def generate():
+        for chunk in text_chunks(text):
+            message = {
+                "id": "chatcmpl-8jqorq6Fw1Vi5XoH7pddGGpQeuPe0",
+                "object": "chat.completion.chunk",
+                "created": int(time.time()),
+                "model": "fay-streaming",
+                "choices": [
+                    {
+                        "delta": {
+                            "content": chunk
+                        },
+                        "index": 0,
+                        "finish_reason": None
+                    }
+                ]
+            }
+            yield f"data: {json.dumps(message)}\n\n"
+            time.sleep(0.1)
+        # 发送最终的结束信号
+        yield 'data: [DONE]\n\n'
+    
+    return Response(generate(), mimetype='text/event-stream')
+
+def non_streaming_response(last_content, text):
+    return jsonify({
+        "id": "chatcmpl-8jqorq6Fw1Vi5XoH7pddGGpQeuPe0",
+        "object": "chat.completion",
+        "created": int(time.time()),
+        "model": "fay",
+        "choices": [
+            {
+                "index": 0,
+                "message": {
+                    "role": "assistant",
+                    "content": text
+                },
+                "logprobs": "",
+                "finish_reason": "stop"
+            }
+        ],
+        "usage": {
+            "prompt_tokens": len(last_content),
+            "completion_tokens": len(text),
+            "total_tokens": len(last_content) + len(text)
+        },
+        "system_fingerprint": "fp_04de91a479"
+    })
+
+def text_chunks(text, chunk_size=20):
+    pattern = r'([^.!?;:，。！？]+[.!?;:，。！？]?)'
+    chunks = re.findall(pattern, text)
+    for chunk in chunks:
+        yield chunk
+
+@__app.route('/', methods=['get'])
+@auth.login_required
+def home_get():
+    return __get_template()
+
+
+@__app.route('/', methods=['post'])
+@auth.login_required
+def home_post():
+    wsa_server.get_web_instance.add_cmd({"is_connect": wsa_server.get_instance().isConnect}) #TODO 不应放这里，同步数字人连接状态
+    return __get_template()
+
+@__app.route('/setting', methods=['get'])
+def setting():
+    return render_template('setting.html')
+
+
+
+#输出的音频http
+@__app.route('/audio/<filename>')
+def serve_audio(filename):
+    audio_file = os.path.join(os.getcwd(), "samples", filename)
+    return send_file(audio_file)
+
+#输出的表情git
+@__app.route('/robot/<filename>')
+def serve_gif(filename):
+    gif_file = os.path.join(os.getcwd(), "gui", "robot", filename)
+    return send_file(gif_file)
+
+def run():
+    server = pywsgi.WSGIServer(('0.0.0.0',5000), __app)
+    server.serve_forever()
+
+def start():
+    MyThread(target=run).start()

gui/static/css/index.css

@@ -0,0 +1,278 @@
+
+html {
+	font-size: 14px;
+}
+  
+  body {
+	background-image: url(../images/Bg_pic.png);
+	background-repeat: repeat-x;
+	margin: 0;
+	font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen', 'Ubuntu', 'Cantarell', 'Fira Sans',
+	  'Droid Sans', 'Helvetica Neue', 'Microsoft Yahei', sans-serif;
+	-webkit-font-smoothing: antialiased;
+	-moz-osx-font-smoothing: grayscale;
+	width: 100%;
+	height: 100vh; 
+	overflow: hidden; 
+  }
+  .main_bg{
+	background-image: url(../images/Mid_bg.png);
+	background-repeat: no-repeat;
+	background-position: center center;
+	width: 100%;
+	margin: 100px auto;
+	max-width: 1719px;
+	min-height:790px;
+	clear: both;
+  }
+  
+  .main_left{
+	float: left;
+	height: 700px;
+	width: 21%;
+	padding-top: 30px;
+  }
+
+  .main_left_logo{
+	height: 200px;
+	text-align: center;
+  }
+
+
+  .main_left_menu{
+	margin-top: 30px;
+	margin-bottom: 80px;
+  }
+  .main_left_menu ul{
+	list-style-type: none;
+    margin: 0;
+    padding-left:9px;
+    width: 352px;	   
+  }
+  .main_left_menu ul li{
+	height: 65px;
+	margin-top: 15px;
+	margin-bottom: 15px;
+	line-height: 52px;
+	font-size: 20px;
+  }
+  .main_left_menu ul li a{ 
+	font-size: 20px;
+	text-align: center; 
+	display: block;
+    color: #555; 
+    text-decoration: none;}
+ 
+  .main_left_menu ul li a:hover {
+	/* background-color: #f9fbff; */
+    color: #0064fb;
+	/* background-image: url('../images/menu_bg_h.png') no-repeat !important; */
+	background-position: center;   
+}
+
+  .changeImg{
+	width: 352px;
+	height: 65px;
+	line-height: 65px;
+	cursor: pointer;
+	
+  }
+  .iconimg1 {
+	background: url('../images/message.png') no-repeat;
+	background-size: 32px;
+	background-position: 100px 50%;
+	display: block;
+	text-align: center;
+  }
+  .iconimg1:hover{	
+	background: url('../images/message_h.png') no-repeat;
+	background-size: 32px;
+	background-position: 100px 50%;
+  }
+  .iconimg2 {
+	background: url('../images/setting.png') no-repeat;
+	background-size: 32px;
+	background-position: 100px 50%;
+	display: block;
+	text-align: center;
+  }
+  .iconimg2:hover{	
+	background: url('../images/setting_h.png') no-repeat;	
+	background-size: 32px;
+	background-position: 100px 50%;
+  }
+  .changeImg:hover{	
+	background: url('../images/menu_bg_h.png') no-repeat;	
+	/* z-index: 10; */
+  }
+  
+  .changeImg2{
+	/* width: 352px; */
+	height: 65px;
+	line-height: 65px;
+	cursor: pointer;
+	/* background: url('../images/menu_bg_h.png') no-repeat; */
+  }
+  .changeImg2:hover{
+	background: url('../images/menu_bg_h.png') no-repeat;
+  }
+  
+  .main_left_emoji{
+	text-align: center;
+	height: 280px; 
+	background-image: url(../images/emoji_bg.png);
+	background-repeat: no-repeat; 
+	background-position: center center;
+ }
+
+  .main_right{float: right;width: 79%;height: 720px;}
+
+.top_info{font-size: 14px; color: #617bab; line-height: 50px; text-align: left;width: 1000px; overflow: hidden; white-space: nowrap; text-overflow: ellipsis; }
+.top_info_text{font-size: 15px;font-weight: bold;}
+.chatmessage{
+	padding-bottom: 48px;  
+    overflow-y: scroll;
+    height: 480px;
+	margin-bottom: 130px;
+	z-index: 1;
+}
+
+
+ 
+.chat-container {
+	max-width: 100%;
+	margin: 20px auto;	
+	padding: 20px;
+  }
+
+.message {
+	display: flex;
+	margin-bottom: 20px;
+  }
+
+.receiver-message {
+	justify-content: flex-start;
+	padding-right: 30%;
+  }
+
+.sender-message {
+	justify-content: flex-end;
+	padding-left: 30%;
+  }
+
+.avatar {
+	width: 52px;
+	height: 52px;
+	border-radius: 50%;
+	margin-right: 15px;
+	margin-left: 15px;
+  }
+
+.message-content {
+	flex:  0 1 auto;
+  }
+
+.message-bubble {
+	background-color: #FFFFFF;
+	border-radius: 6px;
+	padding: 8px;
+	font-size: 15px;
+	color: #333;
+  }
+
+.sender-message.message-bubble {
+	font-size: 15px;
+	padding: 8px;
+	background-color: #008fff;
+	color: #FFFFFF;
+  }
+
+.message-time {
+	font-size: 12px;
+	color: #999;
+	margin-top: 5px;
+	text-align: left;
+  }
+
+.sender-message-time {
+	font-size: 12px;
+	color: #999;
+	margin-top: 5px;
+	text-align: right;
+  }
+
+.Userchange{
+	background-color: #FFFFFF;
+	height: 40px;
+	font-size: 12px;
+}
+
+.inputmessage{
+	margin-left:15% ;
+	width: 760px;
+	background: #f9fbff;
+	border-radius: 70px;
+	height: 73px;
+	box-shadow: -10px 0 15px rgba(0, 0, 0, 0.1);
+	position: absolute;
+    top: 70%;
+	z-index: 2;
+}
+
+.inputmessage_voice{
+	width: 50px;
+	float: left;
+	height: 73px;
+	padding: 15px 5px 0 20px;
+}
+
+.inputmessage_input{
+	background-color: #FFFFFF;
+	width: 540px;
+	float: left;
+	margin-top: 15px;
+	height: 45px;
+}
+
+.inputmessage_send{
+	width: 50px;
+	float: left;
+	height: 73px;
+	padding: 15px 5px 0 15px;
+}
+
+.inputmessage_open{
+	width: 60px;
+	float: right;
+	height: 73px;
+	padding: 15px 5px 0 5px;
+}
+
+.text_in{
+	width: 540px;
+	height: 45px;
+	padding: unset;
+    outline: unset;
+    border-style: unset;
+    background-color: unset;
+    resize: unset;
+	font-size: 14px;
+}
+
+ .tag-container {	background-color: #FFFFFF;
+	display: flex;
+	
+  }
+
+ .tag {	
+	background: url('../images/tabline.png') right no-repeat;
+	padding: 5px 10px;
+	font-size: 14px;
+	cursor: pointer;
+	color: #617bab;line-height: 30px;
+  }
+
+ .tag.selected {
+	background-color: #f4f7ff;
+	color: #0064fb;
+  }

gui/static/css/setting.css

@@ -0,0 +1,42 @@
+
+html {
+	font-size: 14px;
+}
+  
+body {
+	background-image: url(../images/Bg_pic.png);
+	background-repeat: repeat-x;
+	margin: 0;
+	font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen', 'Ubuntu', 'Cantarell', 'Fira Sans',
+	  'Droid Sans', 'Helvetica Neue', 'Microsoft Yahei', sans-serif;
+	-webkit-font-smoothing: antialiased;
+	-moz-osx-font-smoothing: grayscale;
+	width: 100%;
+	height: 100vh; 
+	overflow: hidden; 
+  }
+ 
+
+
+.setting_fay{width: 1200px; padding:10px 25px;height: 390px;}
+.setting_fay_top{font-size: 16px; color: #a3aaaf; line-height: 50px; text-align: left; padding-left: 15px;}
+.setting_name{ float: left;width: 33%;}
+.setting_name ul{list-style-type: none; width: 305px;; }
+.setting_name ul li{ margin: 25px 0; }
+.font_name {font-size: 15px;font-family: Microsoft YaHei;line-height: 21px;color: #2a2a2a;margin-right: 6px;}
+.section_1 {background-color: #ffffff;border-radius: 1px;width: 240px;height: 35px;border: solid 1px #d9d9d9;line-height: 30px;font-size: 16px;}
+
+.setting_work{float: left;width: 31%;}
+.setting_work{ float: left;width: 30%;}
+.setting_work ul{list-style-type: none; width: 330px; }
+.setting_work ul li{ margin: 25px 0; }
+
+.setting_rup{float: right;width: 30%;}
+.setting_wakeup{ width: 330px;}
+.setting_wakeup ul{list-style-type: none; width: 330px;}
+.setting_wakeup ul li{ margin: 25px 0; }
+.microphone{ width: 330px; height: 30px;}
+.microphone_group1{ width: 145px; float: right; }
+
+.setting_autoplay{width: 1086px; background-color: #ffffff; height: 40px; margin-left: 65px; padding:25px 30px; margin-top: 10px;;}
+.section_2 {background-color: #f5f7fa;border-radius: 3px;width: 938px;height: 35px;border: solid 1px #f5f7fa;}

gui/static/js/index.js

@@ -0,0 +1,313 @@
+// fayApp.js
+class FayInterface {
+    constructor(baseWsUrl, baseApiUrl, vueInstance) {
+      this.baseWsUrl = baseWsUrl;
+      this.baseApiUrl = baseApiUrl;
+      this.websocket = null;
+      this.vueInstance = vueInstance; 
+    }
+  
+    connectWebSocket() {
+      if (this.websocket) {
+        this.websocket.onopen = null;
+        this.websocket.onmessage = null;
+        this.websocket.onclose = null;
+        this.websocket.onerror = null;
+      }
+  
+      this.websocket = new WebSocket(this.baseWsUrl);
+  
+      this.websocket.onopen = () => {
+        console.log('WebSocket connection opened');
+      };
+  
+      this.websocket.onmessage = (event) => {
+        const data = JSON.parse(event.data);
+        this.handleIncomingMessage(data);
+      };
+  
+      this.websocket.onclose = () => {
+        console.log('WebSocket connection closed. Attempting to reconnect...');
+        setTimeout(() => this.connectWebSocket(), 5000); 
+      };
+  
+      this.websocket.onerror = (error) => {
+        console.error('WebSocket error:', error);
+      };
+    }
+  
+    async fetchData(url, options = {}) {
+      try {
+        const response = await fetch(url, options);
+        if (!response.ok) throw new Error(`HTTP error! Status: ${response.status}`);
+        return await response.json();
+      } catch (error) {
+        console.error('Error fetching data:', error);
+        return null;
+      }
+    }
+  
+    getVoiceList() {
+      return this.fetchData(`${this.baseApiUrl}/api/get-voice-list`);
+    }
+  
+    getAudioDeviceList() {
+      return this.fetchData(`${this.baseApiUrl}/api/get-audio-device-list`);
+    }
+  
+    submitConfig(config) {
+      return this.fetchData(`${this.baseApiUrl}/api/submit`, {
+        method: 'POST',
+        headers: { 'Content-Type': 'application/json' },
+        body: JSON.stringify({ config })
+      });
+    }
+  
+    controlEyes(state) {
+      return this.fetchData(`${this.baseApiUrl}/api/control-eyes`, {
+        method: 'POST',
+        headers: { 'Content-Type': 'application/json' },
+        body: JSON.stringify({ state })
+      });
+    }
+  
+    startLive() {
+      return this.fetchData(`${this.baseApiUrl}/api/start-live`, {
+        method: 'POST'
+      });
+    }
+  
+    stopLive() {
+      return this.fetchData(`${this.baseApiUrl}/api/stop-live`, {
+        method: 'POST'
+      });
+    }
+  
+    getMessageHistory(username) {
+      return new Promise((resolve, reject) => {
+        const url = `${this.baseApiUrl}/api/get-msg`;
+        const xhr = new XMLHttpRequest();
+        xhr.open("POST", url);
+        xhr.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
+        const send_data = `data=${encodeURIComponent(JSON.stringify({ username }))}`;
+        xhr.send(send_data);
+  
+        xhr.onreadystatechange = function () {
+          if (xhr.readyState === 4) {
+            if (xhr.status === 200) {
+              try {
+                const data = JSON.parse(xhr.responseText);
+                if (data && data.list) {
+                  const combinedList = data.list.flat(); 
+                  resolve(combinedList);
+                } else {
+                  resolve([]);
+                }
+              } catch (e) {
+                console.error('Error parsing response:', e);
+                reject(e);
+              }
+            } else {
+              reject(new Error(`Request failed with status ${xhr.status}`));
+            }
+          }
+        };
+      });
+    }
+  
+    getUserList() {
+      return this.fetchData(`${this.baseApiUrl}/api/get-member-list`, {
+        method: 'POST',
+        headers: { 'Content-Type': 'application/json' }
+      });
+    }
+  
+    handleIncomingMessage(data) {
+      const vueInstance = this.vueInstance; 
+    //   console.log('Incoming message:', data);
+      if (data.liveState !== undefined) {
+        vueInstance.liveState = data.liveState;
+        if (data.liveState === 1) {
+          vueInstance.configEditable = false;
+          vueInstance.sendSuccessMsg('已开启！');
+        } else if (data.liveState === 0) {
+          vueInstance.configEditable = true;
+          vueInstance.sendSuccessMsg('已关闭！');
+        }
+      }
+  
+      if (data.voiceList !== undefined) {
+        vueInstance.voiceList = data.voiceList.map(voice => ({
+          value: voice.id,
+          label: voice.name
+        }));
+      }
+  
+      if (data.deviceList !== undefined) {
+        vueInstance.deviceList = data.deviceList.map(device => ({
+          value: device,
+          label: device
+        }));
+      }
+  
+      if (data.panelMsg !== undefined) {
+        vueInstance.panelMsg = data.panelMsg; 
+      
+      }
+      if (data.robot) {
+        console.log(data.robot)
+        vueInstance.$set(vueInstance, 'robot', data.robot); 
+        }
+      if (data.panelReply !== undefined) {
+        vueInstance.panelReply = data.panelReply.content; 
+        const userExists = vueInstance.userList.some(user => user[1] === data.panelReply.username);
+        if (!userExists) {
+          vueInstance.userList.push([data.panelReply.uid, data.panelReply.username]);
+        }
+        if (vueInstance.selectedUser && data.panelReply.username === vueInstance.selectedUser[1]) {
+          vueInstance.messages.push({
+            username: data.panelReply.username,
+            content: data.panelReply.content,
+            type: data.panelReply.type,
+            time: new Date().toLocaleTimeString()
+          });
+          vueInstance.$nextTick(() => {
+            const chatContainer = vueInstance.$el.querySelector('.chatmessage');
+            if (chatContainer) {
+              chatContainer.scrollTop = chatContainer.scrollHeight;
+            }
+          });
+        }
+      }
+  
+      if (data.is_connect !== undefined) {
+        vueInstance.isConnected = data.is_connect;
+      }
+  
+      if (data.remote_audio_connect !== undefined) {
+        vueInstance.remoteAudioConnected = data.remote_audio_connect;
+      }
+    }
+  }
+  
+  new Vue({
+    el: '#app',
+    delimiters: ["[[", "]]"],
+    data() {
+      return {
+        messages: [],
+        newMessage: '',
+        fayService: null,
+        liveState: 0,
+        isConnected: false,
+        remoteAudioConnected: false,
+        userList: [],
+        selectedUser: null,
+        loading: false,
+        chatMessages: {},
+        panelMsg: '', 
+        panelReply: '', 
+        robot:'static/images/Normal.gif'
+      };
+    },
+    created() {
+      this.initFayService();
+      this.loadUserList();
+    },
+    methods: {
+      initFayService() {
+        this.fayService = new FayInterface('ws://127.0.0.1:10003', 'http://127.0.0.1:5000', this);
+        this.fayService.connectWebSocket();
+      },
+      sendMessage() {
+        let _this = this;
+        let text = _this.newMessage;
+        if (!text) {
+          alert('请输入内容');
+          return;
+        }
+        if (_this.selectedUser === 'others' && !_this.othersUser) {
+          alert('请输入自定义用户名');
+          return;
+        }
+        if (this.liveState != 1) {
+          alert('请先开启服务');
+          return;
+        }
+        let usernameToSend = _this.selectedUser === 'others' ? _this.othersUser : _this.selectedUser[1];
+  
+        this.timer = setTimeout(() => {
+          let height = document.querySelector('.chatmessage').scrollHeight;
+          document.querySelector('.chatmessage').scrollTop = height;
+        }, 1000);
+        _this.newMessage = '';
+        let url = "http://127.0.0.1:5000/api/send";
+        let send_data = {
+          "msg": text,
+          "username": usernameToSend
+        };
+  
+        let xhr = new XMLHttpRequest();
+        xhr.open("post", url);
+        xhr.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
+        xhr.send('data=' + encodeURIComponent(JSON.stringify(send_data)));
+        let executed = false;
+        xhr.onreadystatechange = async function () {
+          if (!executed && xhr.status === 200) {
+            executed = true;
+            // 成功处理逻辑（可以添加额外的回调操作）
+          }
+        };
+      },
+      loadUserList() {
+        this.fayService.getUserList().then((response) => {
+          if (response && response.list) {
+            this.userList = response.list;
+  
+            if (this.userList.length > 0) {
+              this.selectUser(this.userList[0]);
+            }
+          }
+        });
+      },
+      selectUser(user) {
+        this.selectedUser = user;
+        this.fayService.websocket.send(JSON.stringify({ "Username": user[1] }));
+        this.loadMessageHistory(user[1]); 
+      },
+      startLive() {
+        this.liveState = 2
+        this.fayService.startLive().then(() => {
+        });
+    },
+    stopLive() {
+        this.fayService.stopLive().then(() => {
+            this.liveState = 3
+        });
+    },
+
+      loadMessageHistory(username) {
+        this.fayService.getMessageHistory(username).then((response) => {
+          if (response) {
+            this.messages = response;
+            console.log(this.messages);
+            this.$nextTick(() => {
+              const chatContainer = this.$el.querySelector('.chatmessage');
+              if (chatContainer) {
+                chatContainer.scrollTop = chatContainer.scrollHeight;
+              }
+            });
+          }
+        });
+      },
+      sendSuccessMsg(message) {
+        this.$notify({
+            title: '成功',
+            message,
+            type: 'success',
+        });
+    
+}
+    }
+  });
+  

gui/static/js/self-adaption.js

@@ -0,0 +1,25 @@
+window.onload = function () {
+  document.body.style.zoom = "normal";//避免zoom尺寸叠加
+  let scale = document.body.clientWidth / 1920;
+  document.body.style.zoom = scale;
+}; (function () {
+    var throttle = function (type, name, obj) {
+      obj = obj || window;
+      var running = false;
+      var func = function () {
+        if (running) { return; }
+        running = true;
+        requestAnimationFrame(function () {
+          obj.dispatchEvent(new CustomEvent(name));
+          running = false;
+        });
+      };
+      obj.addEventListener(type, func);
+    };
+    throttle("resize", "optimizedResize");
+  })();
+window.addEventListener("optimizedResize", function () {
+  document.body.style.zoom = "normal";
+  let scale = document.body.clientWidth / 1920;
+  document.body.style.zoom = scale;
+});

gui/static/js/setting.js

@@ -0,0 +1,312 @@
+class FayInterface {
+    constructor(baseWsUrl, baseApiUrl, vueInstance) {
+        this.baseWsUrl = baseWsUrl;
+        this.baseApiUrl = baseApiUrl;
+        this.websocket = null;
+        this.vueInstance = vueInstance; 
+    }
+
+    connectWebSocket() {
+        if (this.websocket) {
+            this.websocket.onopen = null;
+            this.websocket.onmessage = null;
+            this.websocket.onclose = null;
+            this.websocket.onerror = null;
+        }
+
+        this.websocket = new WebSocket(this.baseWsUrl);
+
+        this.websocket.onopen = () => {
+            console.log('WebSocket connection opened');
+        };
+
+        this.websocket.onmessage = (event) => {
+            const data = JSON.parse(event.data);
+            this.handleIncomingMessage(data);
+        };
+
+        this.websocket.onclose = () => {
+            console.log('WebSocket connection closed. Attempting to reconnect...');
+            setTimeout(() => this.connectWebSocket(), 5000); 
+        };
+
+        this.websocket.onerror = (error) => {
+            console.error('WebSocket error:', error);
+        };
+    }
+
+    async fetchData(url, options = {}) {
+        try {
+            const response = await fetch(url, options);
+            if (!response.ok) throw new Error(`HTTP error! Status: ${response.status}`);
+            return await response.json();
+        } catch (error) {
+            console.error('Error fetching data:', error);
+            return null;
+        }
+    }
+
+    getData() {
+        return this.fetchData(`${this.baseApiUrl}/api/get-data`, {
+            method: 'POST',
+            headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
+        });
+    }
+
+    submitConfig(config) {
+        return this.fetchData(`${this.baseApiUrl}/api/submit`, {
+            method: 'POST',
+            headers: { 'Content-Type': 'application/json' },
+            body: JSON.stringify({ config }),
+        });
+    }
+
+    startLive() {
+        return this.fetchData(`${this.baseApiUrl}/api/start-live`, {
+            method: 'POST',
+        });
+    }
+
+    stopLive() {
+        return this.fetchData(`${this.baseApiUrl}/api/stop-live`, {
+            method: 'POST',
+        });
+    }
+
+    handleIncomingMessage(data) {
+        const vueInstance = this.vueInstance; 
+        console.log('Incoming message:', data);
+        if (data.liveState !== undefined) {
+            vueInstance.liveState = data.liveState;
+            if (data.liveState === 1) {
+                vueInstance.configEditable = false;
+                vueInstance.sendSuccessMsg('已开启！');
+            } else if (data.liveState === 0) {
+                vueInstance.configEditable = true;
+                vueInstance.sendSuccessMsg('已关闭！');
+            }
+        }
+
+        if (data.voiceList !== undefined) {
+            vueInstance.voiceList = data.voiceList.map((voice) => ({
+                value: voice.id,
+                label: voice.name,
+            }));
+        }
+        if (data.robot) {
+            console.log(data.robot);
+            vueInstance.$set(vueInstance, 'robot', data.robot);
+        }
+
+        if (data.is_connect !== undefined) {
+            vueInstance.isConnected = data.is_connect;
+        }
+
+        if (data.remote_audio_connect !== undefined) {
+            vueInstance.remoteAudioConnected = data.remote_audio_connect;
+        }
+    }
+}
+
+new Vue({
+    el: '#app',
+    delimiters: ['[[', ']]'],
+    data() {
+        return {
+            messages: [],
+            newMessage: '',
+            fayService: null,
+            liveState: 0,
+            isConnected: false,
+            remoteAudioConnected: false,
+            userList: [],
+            selectedUser: null,
+            loading: false,
+            chatMessages: {},
+            panelMsg: '',
+            panelReply: '',
+            robot: 'images/emoji.png',
+            configEditable: true,
+            source_liveRoom_url: '',
+            play_sound_enabled: false,
+            visualization_detection_enabled: false,
+            source_record_enabled: false,
+            source_record_device: '',
+            attribute_name: "",
+            attribute_gender: "",
+            attribute_age: "",
+            attribute_birth: "",
+            attribute_zodiac: "",
+            attribute_constellation: "",
+            attribute_job: "",
+            attribute_hobby: "",
+            attribute_contact: "",
+            attribute_voice: "",
+            QnA:"",
+            interact_perception_gift: 0,
+            interact_perception_follow: 0,
+            interact_perception_join: 0,
+            interact_perception_chat: 0,
+            interact_perception_indifferent: 0,
+            interact_maxInteractTime: 15,
+            voiceList: [],
+            deviceList: [],
+            wake_word_enabled:false,
+            wake_word: '',
+            loading: false,
+            remote_audio_connect: false,
+            wake_word_type: 'common',
+            wake_word_type_options: [{
+                value: 'common',
+                label: '普通'
+            }, {
+                value: 'front',
+                label: '前置词'
+            }],
+            automatic_player_status: false,
+            automatic_player_url: "",
+        };
+    },
+    created() {
+        this.initFayService();
+        this.getData();
+    },
+    methods: {
+        initFayService() {
+            this.fayService = new FayInterface('ws://127.0.0.1:10003', 'http://127.0.0.1:5000', this);
+            this.fayService.connectWebSocket();
+        },
+        getData() {
+            this.fayService.getData().then((data) => {
+                if (data) {
+                    this.voiceList =  data.voice_list.map((voice) => ({
+                        value: voice.id,
+                        label: voice.name,
+                    }));
+                    this.updateConfigFromData(data.config);
+                }
+            });
+        },
+        updateConfigFromData(config) {
+          
+            if (config.interact) {
+                this.play_sound_enabled = config.interact.playSound;
+                this.visualization_detection_enabled = config.interact.visualization;
+                this.QnA = config.interact.QnA;
+            }
+            if (config.source && config.source.record) {
+                this.source_record_enabled = config.source.record.enabled;
+                this.source_record_device = config.source.record.device;
+                this.wake_word = config.source.wake_word;
+                this.wake_word_type = config.source.wake_word_type;
+                this.wake_word_enabled = config.source.wake_word_enabled;
+                this.automatic_player_status = config.source.automatic_player_status;
+                this.automatic_player_url = config.source.automatic_player_url;
+
+            }
+            if (config.attribute) {
+                this.attribute_name = config.attribute.name;
+                this.attribute_gender = config.attribute.gender;
+                this.attribute_age = config.attribute.age;
+                this.attribute_name = config.attribute.name;
+                this.attribute_gender = config.attribute.gender;
+                this.attribute_birth = config.attribute.birth;
+                this.attribute_zodiac = config.attribute.zodiac;
+                this.attribute_constellation = config.attribute.constellation;
+                this.attribute_job = config.attribute.job;
+                this.attribute_hobby = config.attribute.hobby;
+                this.attribute_contact = config.attribute.contact;
+                this.attribute_voice = config.attribute.voice;
+
+            }
+            if (config.interact.perception) {
+                this.interact_perception_follow = config.interact.perception.follow;
+            }
+        },
+        saveConfig() {
+            let url = "http://127.0.0.1:5000/api/submit";
+            let send_data = {
+                "config": {
+                    "source": {
+                        "liveRoom": {
+                            "enabled": this.configEditable,
+                            "url": this.source_liveRoom_url
+                        },
+                        "record": {
+                            "enabled": this.source_record_enabled,
+                            "device": this.source_record_device
+                        },
+                        "wake_word_enabled": this.wake_word_enabled,
+                        "wake_word": this.wake_word,
+                        "wake_word_type": this.wake_word_type,
+                        "tts_enabled": this.tts_enabled,
+                        "automatic_player_status": this.automatic_player_status,
+                        "automatic_player_url": this.automatic_player_url
+                    },
+                    "attribute": {
+                        "voice": this.attribute_voice,
+                        "name": this.attribute_name,
+                        "gender": this.attribute_gender,
+                        "age": this.attribute_age,
+                        "birth": this.attribute_birth,
+                        "zodiac": this.attribute_zodiac,
+                        "constellation": this.attribute_constellation,
+                        "job": this.attribute_job,
+                        "hobby": this.attribute_hobby,
+                        "contact": this.attribute_contact
+                    },
+                    "interact": {
+                        "playSound": this.play_sound_enabled,
+                        "visualization": this.visualization_detection_enabled,
+                        "QnA": this.QnA,
+                        "maxInteractTime": this.interact_maxInteractTime,
+                        "perception": {
+                            "gift": this.interact_perception_follow,
+                            "follow": this.interact_perception_follow,
+                            "join": this.interact_perception_follow,
+                            "chat": this.interact_perception_follow,
+                            "indifferent": this.interact_perception_follow 
+                        }
+                    },
+                    "items": []
+                }
+            };
+
+            let xhr = new XMLHttpRequest()
+            xhr.open("post", url)
+            xhr.setRequestHeader("Content-type", "application/x-www-form-urlencoded")
+            xhr.send('data=' + JSON.stringify(send_data))
+            let executed = false
+            xhr.onreadystatechange = async function () {
+                if (!executed && xhr.status === 200) {
+                    try {
+                        let data = await eval('(' + xhr.responseText + ')')
+                        console.log("data: " + data['result'])
+                        executed = true
+                    } catch (e) {
+                    }
+                }
+            }
+            this.sendSuccessMsg("配置已保存！")
+        },
+        startLive() {
+            this.liveState = 2
+            this.fayService.startLive().then(() => {
+                this.configEditable = false;
+            });
+        },
+        stopLive() {
+            this.fayService.stopLive().then(() => {
+                this.configEditable = true;
+                this.liveState = 3
+            });
+        },
+        sendSuccessMsg(message) {
+            this.$notify({
+                title: '成功',
+                message,
+                type: 'success',
+            });
+        },
+    },
+});

gui/static/live2d/autoload.js

@@ -0,0 +1,62 @@
+// live2d_path 参数建议使用绝对路径
+const live2d_path = "https://fastly.jsdelivr.net/gh/stevenjoezhang/live2d-widget@latest/";
+//const live2d_path = "/live2d-widget/";
+
+// 封装异步加载资源的方法
+function loadExternalResource(url, type) {
+	return new Promise((resolve, reject) => {
+		let tag;
+
+		if (type === "css") {
+			tag = document.createElement("link");
+			tag.rel = "stylesheet";
+			tag.href = url;
+		}
+		else if (type === "js") {
+			tag = document.createElement("script");
+			tag.src = url;
+		}
+		if (tag) {
+			tag.onload = () => resolve(url);
+			tag.onerror = () => reject(url);
+			document.head.appendChild(tag);
+		}
+	});
+}
+
+// 加载 waifu.css live2d.min.js waifu-tips.js
+if (screen.width >= 768) {
+	Promise.all([
+		loadExternalResource(live2d_path + "waifu.css", "css"),
+		loadExternalResource(live2d_path + "live2d.min.js", "js"),
+		loadExternalResource(live2d_path + "waifu-tips.js", "js")
+	]).then(() => {
+		// 配置选项的具体用法见 README.md
+		initWidget({
+			waifuPath:  "/static/live2d/waifu-tips.json",
+			//apiPath: "https://live2d.fghrsh.net/api/",
+			cdnPath: "https://fastly.jsdelivr.net/gh/fghrsh/live2d_api/",
+			tools: ["switch-model", "quit"]
+		});
+	});
+}
+
+console.log(`
+  く__,.ヘヽ.        /  ,ー､ 〉
+           ＼ ', !-─‐-i  /  /´
+           ／｀ｰ'       L/／｀ヽ､
+         /   ／,   /|   ,   ,       ',
+       ｲ   / /-‐/  ｉ  L_ ﾊ ヽ!   i
+        ﾚ ﾍ 7ｲ｀ﾄ   ﾚ'ｧ-ﾄ､!ハ|   |
+          !,/7 '0'     ´0iソ|    |
+          |.从"    _     ,,,, / |./    |
+          ﾚ'| i＞.､,,__  _,.イ /   .i   |
+            ﾚ'| | / k_７_/ﾚ'ヽ,  ﾊ.  |
+              | |/i 〈|/   i  ,.ﾍ |  i  |
+             .|/ /  ｉ：    ﾍ!    ＼  |
+              kヽ>､ﾊ    _,.ﾍ､    /､!
+              !'〈//｀Ｔ´', ＼ ｀'7'ｰr'
+              ﾚ'ヽL__|___i,___,ンﾚ|ノ
+                  ﾄ-,/  |___./
+                  'ｰ'    !_,.:
+`);

gui/static/live2d/waifu-tips.json

@@ -0,0 +1,120 @@
+{
+	"mouseover": [{
+		"selector": "#live2d",
+		"text": ["干嘛呢你，快把手拿开～～", "鼠…鼠标放错地方了！", "你要干嘛呀？", "喵喵喵？", "怕怕(ノ≧∇≦)ノ", "非礼呀！救命！", "这样的话，只能使用武力了！", "我要生气了哦", "不要动手动脚的！", "真…真的是不知羞耻！", "Hentai！"]
+	}, {
+		"selector": "#waifu-tool-hitokoto",
+		"text": ["猜猜我要说些什么？", "我从青蛙王子那里听到了不少人生经验。"]
+	}, {
+		"selector": "#waifu-tool-asteroids",
+		"text": ["要不要来玩飞机大战？", "这个按钮上写着「不要点击」。", "怎么，你想来和我玩个游戏？", "听说这样可以蹦迪！"]
+	}, {
+		"selector": "#waifu-tool-switch-model",
+		"text": ["你是不是不爱人家了呀，呜呜呜～", "要见见我的姐姐嘛？", "想要看我妹妹嘛？", "要切换看板娘吗？"]
+	}, {
+		"selector": "#waifu-tool-switch-texture",
+		"text": ["喜欢换装 PLAY 吗？", "这次要扮演什么呢？", "变装！", "让我们看看接下来会发生什么！"]
+	}, {
+		"selector": "#waifu-tool-photo",
+		"text": ["你要给我拍照呀？一二三～茄子～", "要不，我们来合影吧！", "保持微笑就好了～"]
+	}, {
+		"selector": "#waifu-tool-info",
+		"text": ["想要知道更多关于我的事么？", "这里记录着我搬家的历史呢。", "你想深入了解我什么呢？"]
+	}, {
+		"selector": "#waifu-tool-quit",
+		"text": ["到了要说再见的时候了吗？", "呜呜 QAQ 后会有期……", "不要抛弃我呀……", "我们，还能再见面吗……", "哼，你会后悔的！"]
+	}, {
+		"selector": ".character_left",
+		"text": ["这是我的人设，修改之后记得重新启动哦"]
+	}, {
+		"selector": ".character_right",
+		"text": ["这是我的性格，修改之后记得重新启动哦"]
+	}, {
+		"selector": ".right_main",
+		"text": ["我可以做你的售货员了"]
+	}, {
+		"selector": ".btn_close",
+		"text": ["你是不想和我说话了吗"]
+	}, {
+		"selector": ".btn_open",
+		"text": ["快点击，我可以和你谈谈心了"]
+	}],
+	"click": [{
+		"selector": "#live2d",
+		"text": ["是…是不小心碰到了吧…", "萝莉控是什么呀？", "你看到我的小熊了吗？", "再摸的话我可要报警了！⌇●﹏●⌇", "110 吗，这里有个变态一直在摸我(ó﹏ò｡)", "不要摸我了，我会告诉老婆来打你的！", "干嘛动我呀！小心我咬你！", "别摸我，有什么好摸的！"]
+	},{
+		"selector": ".btn_close",
+		"text": ["再见！"]
+	}, {
+		"selector": ".btn_open",
+		"text": ["我要上线了"]
+	}],
+	"seasons": [{
+		"date": "01/01",
+		"text": "<span>元旦</span>了呢，新的一年又开始了，今年是{year}年～"
+	}, {
+		"date": "02/14",
+		"text": "又是一年<span>情人节</span>，{year}年找到对象了嘛～"
+	}, {
+		"date": "03/08",
+		"text": "今天是<span>国际妇女节</span>！"
+	}, {
+		"date": "03/12",
+		"text": "今天是<span>植树节</span>，要保护环境呀！"
+	}, {
+		"date": "04/01",
+		"text": "悄悄告诉你一个秘密～<span>今天是愚人节，不要被骗了哦～</span>"
+	}, {
+		"date": "05/01",
+		"text": "今天是<span>五一劳动节</span>，计划好假期去哪里了吗～"
+	}, {
+		"date": "06/01",
+		"text": "<span>儿童节</span>了呢，快活的时光总是短暂，要是永远长不大该多好啊…"
+	}, {
+		"date": "09/03",
+		"text": "<span>中国人民抗日战争胜利纪念日</span>，铭记历史、缅怀先烈、珍爱和平、开创未来。"
+	}, {
+		"date": "09/10",
+		"text": "<span>教师节</span>，在学校要给老师问声好呀～"
+	}, {
+		"date": "10/01",
+		"text": "<span>国庆节</span>到了，为祖国母亲庆生！"
+	}, {
+		"date": "11/05-11/12",
+		"text": "今年的<span>双十一</span>是和谁一起过的呢～"
+	}, {
+		"date": "12/20-12/31",
+		"text": "这几天是<span>圣诞节</span>，主人肯定又去剁手买买买了～"
+	}],
+	"time": [{
+		"hour": "6-7",
+		"text": "早上好！一日之计在于晨，美好的一天就要开始了～"
+	}, {
+		"hour": "8-11",
+		"text": "上午好！工作顺利嘛，不要久坐，多起来走动走动哦！"
+	}, {
+		"hour": "12-13",
+		"text": "中午了，工作了一个上午，现在是午餐时间！"
+	}, {
+		"hour": "14-17",
+		"text": "午后很容易犯困呢，今天的运动目标完成了吗？"
+	}, {
+		"hour": "18-19",
+		"text": "傍晚了！窗外夕阳的景色很美丽呢，最美不过夕阳红～"
+	}, {
+		"hour": "20-21",
+		"text": "晚上好，今天过得怎么样？"
+	}, {
+		"hour": "22-23",
+		"text": ["已经这么晚了呀，早点休息吧，晚安～", "深夜时要爱护眼睛呀！"]
+	}, {
+		"hour": "0-5",
+		"text": "你是夜猫子呀？这么晚还不睡觉，明天起的来嘛？"
+	}],
+	"message": {
+		"default": ["好久不见，日子过得好快呢……", "大坏蛋！你都多久没理人家了呀，嘤嘤嘤～", "嗨～快来逗我玩吧！", "拿小拳拳锤你胸口！", "记得把小家加入收藏夹哦！"],
+		"console": "哈哈，你打开了控制台，是想要看看我的小秘密吗？",
+		"copy": "你都复制了些什么呀，转载要记得加上出处哦！",
+		"visibilitychange": "哇，你终于回来了～"
+	}
+}

gui/templates/index.html

@@ -0,0 +1,107 @@
+<!DOCTYPE html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <title>Fay数字人</title>
+    <link rel="stylesheet" href="{{ url_for('static',filename='css/index.css') }}" />
+    <script src="{{ url_for('static',filename='js/vue.js') }}"></script>
+    <script src="https://cdn.jsdelivr.net/npm/element-ui@2.15.6/lib/index.js"></script>
+    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/element-ui@2.15.6/lib/theme-chalk/index.css" />
+    <script src="{{ url_for('static',filename='js/index.js') }}" defer></script>
+  </head>
+  <body >
+    <div  id="app" class="main_bg">  
+      <div class="main_left">  
+        <div class="main_left_logo" ><img   src="{{ url_for('static',filename='images/logo.png') }}" alt=""> 
+        </div>
+
+      <div class="main_left_menu">
+        <ul>
+          <li class="changeImg"><a href="/"><span class="iconimg1">消息</span></a></li>
+          <li class="changeImg2"><a href="/setting"><span class="iconimg2">设置</span></a></li>
+        </ul>
+      </div>
+      
+      <div class="main_left_emoji"><img style="padding-top: 60px;    max-width: 140px;"  :src="robot" alt="" >
+        
+      </div>
+      </div>
+
+      <div class="main_right">
+        <div class="top_info"><span  class="top_info_text">消息：</span>[[panelMsg]]</div>
+        <!-- 以上是即时信息显示 -->
+        <div class="chatmessage">
+          <div class="chat-container" id="user0" >
+            <div  v-for="(item, index) in messages" :key="index" >
+
+              <div class="message receiver-message" v-if="item.type == 'fay'">
+                <img class="avatar" src="{{ url_for('static',filename='images/Fay_send.png') }}" alt="接收者头像">
+                <div class="message-content">
+                  <div class="message-bubble">[[item.content]]</div>
+                  <div class="message-time">[[item.timetext]]</div>
+                </div>
+              </div>     
+
+            <div class="message sender-message" v-else>
+              <div class="message-content">
+                <div class="sender-message message-bubble">[[item.content]]</div>
+                <div class="sender-message-time">[[item.timetext]]</div>
+              </div>
+              <img class="avatar" src="{{ url_for('static',filename='images/User_send.png') }}" alt="发送者头像">
+            </div>
+          </div>
+          <div >
+                 
+          </div>
+          
+          </div>
+        </div>
+
+        <!-- 以上是聊天对话 -->
+
+        <div class="inputmessage">
+          <div class="inputmessage_voice" ><img  src="{{ url_for('static',filename='images/voice.png') }}" alt="" style="filter: grayscale(100%);" ></div>
+          <div class="inputmessage_input"> <textarea class="text_in" placeholder="请输入内容" v-model="newMessage" @keyup.enter="sendMessage"></textarea></div>
+          <div class="inputmessage_send"><img  src="{{ url_for('static',filename='images/send.png') }}" alt="发送信息" @click="sendMessage"></div>
+
+          <div class="inputmessage_open">
+
+            <img v-if="liveState == 1" src="{{ url_for('static',filename='images/close.png') }}"  @click=stopLive() >
+            <img v-else src="{{ url_for('static',filename='images/open.png') }}"  @click=startLive() >
+          
+          </div>
+
+        </div>
+
+        <div class="Userchange">
+          <div class="tag-container">
+            <div class="tag" v-for="user in userList" :key="user[0]" :class="{'selected': selectedUser && selectedUser[0] === user[0]}" @click="selectUser(user)">
+              [[ user[1] ]]
+            </div>
+          </div>
+        </div>
+
+
+</div>
+<!-- 以上是多用户切换 -->
+
+      </div>
+
+
+
+
+
+
+
+
+    </div>
+
+
+
+
+
+
+
+    </body>

gui/templates/setting.html

@@ -0,0 +1,135 @@
+<!DOCTYPE html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <title>Fay数字人</title>
+    <link rel="stylesheet" href="{{ url_for('static',filename='css/index.css') }}"/>
+    <link rel="stylesheet" href="{{ url_for('static',filename='css/setting.css') }}"/>
+    <!-- 引入样式 -->
+<link rel="stylesheet" href="{{ url_for('static',filename='css/element/index.css') }}">
+  </head>
+
+  <body>
+    <div class="main_bg" id="app">	
+      <div class="main_left">	
+        <div class="main_left_logo" ><img src="{{ url_for('static',filename='images/logo.png') }}" alt=""> 
+        </div>
+
+      <div class="main_left_menu">
+        <ul>
+          <li class="changeImg"><a href="/"><span class="iconimg1">消息</span></a></li>
+          <li class="changeImg2"><a href="/setting"><span class="iconimg2">设置</span></a></li>
+        </ul>
+      </div>
+      
+      <div class="main_left_emoji"><img  src="{{ url_for('static',filename='images/emoji.png') }}" alt="" >
+      </div>
+      </div>
+
+      <div class="main_right">
+        <div class="setting_fay_top"><span style="color: #000; font-size: 20px; padding-right: 10px;">人设</span>请设置你的数字人人设 </div>
+        <!-- 以上是即时信息显示 -->
+  
+        <div class="setting_fay">          
+          <div class="setting_name">
+            <ul>
+              <li> <span class="font_name">姓&nbsp;&nbsp;&nbsp;名:</span><input class="section_1" :disabled="!configEditable" v-model="attribute_name" placeholder="请输入内容" ></li>
+              <li> <span class="font_name">性&nbsp;&nbsp;&nbsp;别:</span><input class="section_1" :disabled="!configEditable" v-model="attribute_gender" placeholder="请输入内容" /></li>
+              <li> <span class="font_name">年&nbsp;&nbsp;&nbsp;龄:</span><input class="section_1" :disabled="!configEditable"  v-model="attribute_age" placeholder="请输入内容" /></li>
+              <li> <span class="font_name">出生地:</span><input class="section_1" :disabled="!configEditable"  v-model="attribute_birth" placeholder="请输入内容" /></li>                    
+              <li> <span class="font_name">生&nbsp;&nbsp;&nbsp;肖:</span><input class="section_1" :disabled="!configEditable"  v-model="attribute_zodiac" placeholder="请输入内容" /></li>
+              <li> <span class="font_name">星&nbsp;&nbsp;&nbsp;座:</span><input class="section_1" :disabled="!configEditable"  v-model="attribute_constellation" placeholder="请输入内容" /></li>  
+            </ul>
+          </div>
+          <div class="setting_work">
+            <ul>
+              <li> <span class="font_name">职&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;业:</span><input class="section_1" :disabled="!configEditable"  v-model="attribute_job" placeholder="请输入内容"/></li>
+              <li> <span class="font_name">&nbsp;联系方式:</span><input class="section_1" :disabled="!configEditable" v-model="attribute_contact" placeholder="请输入内容" /></li>
+              <li> <span class="font_name">Q&A文件:</span><input class="section_1" :disabled="!configEditable" v-model="QnA" placeholder="请输入内容" /></li>
+              <li> <span class="font_name">补&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;充:</span><textarea  class="section_1" style="height: 165px;vertical-align: text-top;"></textarea></li>           
+            </ul>
+          </div>
+          <div class="setting_rup">      
+          <div class="setting_wakeup">            
+            <ul>
+              <li> <span class="font_name">唤醒模式:</span>
+                <el-switch :disabled="!configEditable"
+                v-model="wake_word_enabled"
+                active-color="#13ce66"
+                inactive-color="#ff4949">
+              </el-switch>              
+              </li>
+              <li> <span class="font_name">&nbsp;唤&nbsp;醒&nbsp;词:</span>
+                <input v-model="wake_word" placeholder="请输入内容（以,隔开）" :disabled="!configEditable" class="section_1" /></li>
+              <li> <span class="font_name">唤醒方式:</span>
+                <select class="section_1" v-model="wake_word_type" :disabled="!configEditable">
+                  <option v-for="item in wake_word_type_options" :key="item.value"
+                  :label="item.label" :value="item.value">
+              </option>
+              </select></li>   
+            </ul>
+          </div>
+
+             <div class="microphone"> 
+              <div class="microphone_group1">
+                <span class="font_name">扬&nbsp;声&nbsp;器&nbsp;:</span>
+                <el-switch  v-model="play_sound_enabled" active-color="#13ce66" inactive-color="#ff4949" :disabled="!configEditable"> </el-switch>
+              </div>
+              <div class="microphone_group1" >
+                <span class="font_name">&nbsp;麦&nbsp;克&nbsp;风&nbsp;:</span>
+                <el-switch   v-model="source_record_enabled" active-color="#13ce66" inactive-color="#ff4949" :disabled="!configEditable"> </el-switch>
+              </div>
+            </div> 
+
+            <div class="setting_wakeup">            
+              <ul>
+              <li> <span class="font_name">声音选择:</span>
+                <select v-model="attribute_voice" placeholder="请选择" class="section_1" style="height: 39px;" :disabled="!configEditable">
+                  <option v-for="item in voiceList" :key="item.value" 
+                  :label="item.label" :value="item.value">
+              </option>
+              </select></li>
+              <li style="display: flex;"> <span class="font_name" style="line-height: 36px;">&nbsp;敏&nbsp;感&nbsp;度&nbsp;:</span>
+                <el-slider  style="width: 230px;" v-model="interact_perception_follow" :disabled="!configEditable"></el-slider></li>           
+            </ul>
+          </div>
+        </div>
+
+        </div>
+
+        <div class="setting_autoplay">
+          <div><el-switch
+            v-model="automatic_player_status" @change=saveConfig()
+            active-color="#13ce66" 
+            inactive-color="#ff4949">
+          </el-switch><span class="font_name" style="margin-left: 10px;">自动播放：</span>
+          <input class="section_2" v-model="automatic_player_url" placeholder="http://127.0.0.1:6000" :disabled="!configEditable" />
+        </div>
+        </div>
+        <div style="margin-top: 40px; text-align: center;">
+          <el-button style="width: 160px;height: 50px;margin-left: -100px;" :disabled="!configEditable"  @click=saveConfig()>保存配置</el-button>
+
+          
+          <!-- <el-button type="primary" style="width: 160px;height: 50px;margin-left: 100px;background-color: #007aff;">开 启</el-button> -->
+          <el-button v-if="liveState == 1" type="success" class="btn_close"
+          style="width: 160px;height: 50px;margin-left: 100px;background-color: #007aff;" @click=stopLive()>关闭（运行中）</el-button>
+                                    <el-button v-else-if="liveState == 2" type="primary" plain disabled
+                                    style="width: 160px;height: 50px;margin-left: 100px;background-color: #007aff;">正在开启...</el-button>
+                                    <el-button v-else-if="liveState == 3" type="success" plain disabled
+                                    style="width: 160px;height: 50px;margin-left: 100px;background-color: #007aff;">正在关闭...</el-button>
+                                    <el-button v-else type="primary" style="width: 160px;height: 50px;margin-left: 100px;background-color: #007aff;"
+                                        @click=startLive()>开启</el-button>
+        </div>
+      </div>
+
+    </div>
+
+    </body>
+    <!-- import Vue before Element -->
+    <script src="{{ url_for('static',filename='js/vue.js') }}"></script>
+    <script src="{{ url_for('static',filename='js/element.js') }}"></script> <!-- 这里需要确保先引入 Element -->
+    <script src="{{ url_for('static',filename='js/setting.js') }}"></script>
+ 
+</html>

gui/window.py

@@ -0,0 +1,84 @@
+import os
+
+import time
+
+from PyQt5.QtWidgets import *
+from PyQt5.QtWidgets import QDialog, QHBoxLayout, QVBoxLayout
+from PyQt5.QtWidgets import QGroupBox
+from PyQt5.QtWebEngineWidgets import *
+from PyQt5.QtCore import *
+from PyQt5 import QtWidgets
+
+from scheduler.thread_manager import MyThread
+
+
+class MainWindow(QMainWindow):
+    SigSendMessageToJS = pyqtSignal(str)
+
+    def __init__(self):
+        super(MainWindow, self).__init__()
+        # self.setWindowFlags(Qt.WindowType.WindowShadeButtonHint)
+        self.setWindowTitle('FeiFei Alpha')
+        # self.setFixedSize(16 * 80, 9 * 80) 
+        self.setGeometry(0, 0, 16 * 70, 9 * 70)
+        self.showMaximized()
+        # self.center()
+        self.browser = QWebEngineView()
+        #清空缓存
+        profile = QWebEngineProfile.defaultProfile()
+        profile.clearHttpCache()  
+        self.browser.load(QUrl('http://127.0.0.1:5000'))
+        self.setCentralWidget(self.browser)
+        MyThread(target=self.runnable).start()
+
+    def runnable(self):
+        while True:
+            if not self.isVisible():
+                # try:
+                #     wsa_server.get_instance().stop_server()
+                #     wsa_server.get_web_instance().stop_server()
+                #     thread_manager.stopAll()
+                # except BaseException as e:
+                #     print(e)
+                os.system("taskkill /F /PID {}".format(os.getpid()))
+            time.sleep(0.05)
+
+    def center(self):
+        screen = QtWidgets.QDesktopWidget().screenGeometry()
+        size = self.geometry()
+        self.move((screen.width() - size.width()) / 2, (screen.height() - size.height()) / 2)
+
+    def keyPressEvent(self, event):
+        pass
+        # if event.key() == Qt.Key_F12:
+        #     self.s = TDevWindow()
+        #     self.s.show()
+        #     self.browser.page().setDevToolsPage(self.s.mpJSWebView.page())
+
+    def OnReceiveMessageFromJS(self, strParameter):
+        if not strParameter:
+            return
+
+
+class TDevWindow(QDialog):
+    def __init__(self):
+        super(TDevWindow, self).__init__()
+        self.init_ui()
+
+    def init_ui(self):
+        self.mpJSWebView = QWebEngineView(self)
+        self.url = 'https://www.baidu.com/'
+        self.mpJSWebView.page().load(QUrl(self.url))
+        self.mpJSWebView.show()
+
+        self.pJSTotalVLayout = QVBoxLayout()
+        self.pJSTotalVLayout.setSpacing(0)
+        self.pJSTotalVLayout.addWidget(self.mpJSWebView)
+        self.pWebGroup = QGroupBox('Web View', self)
+        self.pWebGroup.setLayout(self.pJSTotalVLayout)
+
+        self.mainLayout = QHBoxLayout()
+        self.mainLayout.setSpacing(5)
+        self.mainLayout.addWidget(self.pWebGroup)
+        self.setLayout(self.mainLayout)
+        self.setMinimumSize(800, 800)

llm/VllmGPT.py

@@ -0,0 +1,56 @@
+import json
+import requests
+# from core import content_db
+
+class VllmGPT:
+
+    def __init__(self, host="127.0.0.1",
+                 port="8000",
+                 model="THUDM/chatglm3-6b",
+                 max_tokens="1024"):
+        self.host = host
+        self.port = port
+        self.model=model
+        self.max_tokens=max_tokens
+        self.__URL = "http://{}:{}/v1/completions".format(self.host, self.port)
+        self.__URL2 = "http://{}:{}/v1/chat/completions".format(self.host, self.port)
+
+    def question(self,cont):
+        chat_list = []
+        url = "http://127.0.0.1:8101/v1/completions"
+        req = json.dumps({
+            "model": "THUDM/chatglm3-6b",
+            "prompt": cont,
+            "max_tokens": 768,
+            "temperature": 0})
+        print(url)
+        print(req)
+        
+        headers = {'content-type': 'application/json'}
+        r = requests.post(url, headers=headers, data=req)
+        res = json.loads(r.text)
+        
+        return res['choices'][0]['text']
+
+    def question2(self,cont):
+        chat_list = []
+        current_chat={"role": "user", "content": cont}
+        chat_list.append(current_chat)
+        content = {
+            "model": self.model,
+            "messages": chat_list,
+            "max_tokens": 768,
+            "temperature": 0.3,
+            "user":"live-virtual-digital-person"}
+        url = self.__URL2
+        req = json.dumps(content)
+        headers = {'content-type': 'application/json', 'Authorization': 'Bearer '}
+        r = requests.post(url, headers=headers, json=content)
+        res = json.loads(r.text)
+        
+        return res['choices'][0]['message']['content']
+    
+if __name__ == "__main__":
+    vllm = VllmGPT('127.0.0.1','8101','Qwen-7B-Chat')
+    req = vllm.question2("你叫什么名字啊今年多大了")
+    print(req)

llm/nlp_ChatGLM3.py

@@ -0,0 +1,36 @@
+import json
+import requests
+from core import content_db
+
+
+def question(cont, uid=0):
+    contentdb = content_db.new_instance()
+    if uid == 0:
+        list = contentdb.get_list('all','desc', 11)
+    else:
+        list = contentdb.get_list('all','desc', 11, uid)
+    answer_info = dict()
+    chat_list = []
+    i = len(list)-1
+    while i >= 0:
+        answer_info = dict()
+        if list[i][0] == "member":
+            answer_info["role"] = "user"
+            answer_info["content"] = list[i][2]
+        elif list[i][0] == "fay":
+            answer_info["role"] = "bot"
+            answer_info["content"] = list[i][2]
+        chat_list.append(answer_info)
+        i -= 1
+    content = {
+        "prompt":"请简单回复我。" +  cont,
+        "history":chat_list}
+    url = "http://127.0.0.1:8000/v1/completions"
+    req = json.dumps(content)
+    headers = {'content-type': 'application/json'}
+    r = requests.post(url, headers=headers, data=req)
+    res = json.loads(r.text).get('response')
+    return req
+
+if __name__ == "__main__":
+    question("你叫什么名字")

llm/nlp_VisualGLM.py

@@ -0,0 +1,37 @@
+"""
+这是对于清华智谱VisualGLM-6B的代码，在使用前请先安装并启动好VisualGLM-6B.
+https://github.com/THUDM/VisualGLM-6B
+"""
+import json
+import requests
+import uuid
+import os
+import cv2
+from ai_module import yolov8
+
+# Initialize an empty history list
+communication_history = []
+
+def question(cont, uid=0):
+    if not yolov8.new_instance().get_status():
+        return "请先启动“Fay Eyes”"
+    content = {
+        "text":cont,
+        "history":communication_history}
+    img = yolov8.new_instance().get_img()
+    if yolov8.new_instance().get_status() and img is not None:
+        filename = str(uuid.uuid4()) + ".jpg"
+        current_working_directory = os.getcwd()
+        filepath = os.path.join(current_working_directory, "data", filename)
+        cv2.imwrite(filepath, img)
+        content["image"] = filepath
+    url = "http://127.0.0.1:8080"
+    print(content)
+    req = json.dumps(content)
+    headers = {'content-type': 'application/json'}
+    r = requests.post(url, headers=headers, data=req)
+
+    # Save this conversation to history
+    communication_history.append([cont, r.text])
+
+    return r.text + "\n(相片:" + filepath + ")"

llm/nlp_coze.py

@@ -0,0 +1,75 @@
+
+import requests
+import json
+from utils import util
+from utils import config_util as cfg
+from core import content_db
+
+def question(cont, uid=0):
+    contentdb = content_db.new_instance()
+    if uid == 0:
+        communication_history = contentdb.get_list('all','desc', 11)
+    else:
+        communication_history = contentdb.get_list('all','desc', 11, uid)
+    message = []
+    i = len(communication_history) - 1
+
+    if len(communication_history)>1:
+        while i >= 0:
+            answer_info = dict()
+            if communication_history[i][0] == "member":
+                answer_info["role"] = "user"
+                answer_info["type"] = "query"
+                answer_info["content"] = communication_history[i][2]
+                answer_info["content_type"] = "text"
+            elif communication_history[i][0] == "fay":
+                answer_info["role"] = "assistant"
+                answer_info["type"] = "answer"
+                answer_info["content"] = communication_history[i][2]
+                answer_info["content_type"] = "text"
+            message.append(answer_info)
+            i -= 1
+
+    message.append({
+                "role": "user",
+                "content": cont,
+                "content_type": "text"
+            })
+    url = "https://api.coze.cn/v3/chat"
+    payload = json.dumps({
+        "bot_id": cfg.coze_bot_id,
+        "user_id": f"{uid}",
+        "stream": True,
+        "auto_save_history": True,
+        "additional_messages": message
+    })
+    headers = {
+        'Authorization': f"Bearer {cfg.coze_api_key}",
+        'Content-Type': 'application/json'
+    }
+
+    response = requests.post(url, headers=headers, data=payload, stream=True)
+
+    if response.status_code == 200:
+        response_text = ""
+        start = False
+        for line in response.iter_lines():
+            if line:
+                line = line.decode('utf-8')  
+                if line == "event:conversation.message.completed":
+                    start = True
+                if line == "event:done":
+                    return response_text 
+                if start and line.startswith('data:'):
+                    json_str = line[5:]  
+                    try:
+                        event_data = json.loads(json_str)
+                        if event_data.get('type') == 'answer':
+                            response_text = event_data.get('content', '')
+                    except json.JSONDecodeError as e:
+                        print(f"JSON decode error: {e}")
+                        continue
+    else:
+        print(f"调用失败，状态码：{response.status_code}")
+        return "抱歉，我现在太忙了，休息一会，请稍后再试。"
+

llm/nlp_gpt.py

@@ -0,0 +1,99 @@
+"""
+此代码由fay开源开发者社区 江湖墨明 提供
+通过此代码的修改，可以实现对接本地clash代理或远程代理，clash无需设置成系统代理。以解决在开系统代理后无法使用部分功能的问题
+"""
+
+import requests
+import time
+
+from utils import config_util as cfg
+from urllib3.exceptions import InsecureRequestWarning
+requests.packages.urllib3.disable_warnings(category=InsecureRequestWarning)
+from core import content_db
+from utils import util
+import json
+
+httpproxy = cfg.proxy_config
+
+def question(cont, uid=0):
+    url= cfg.gpt_base_url + "/chat/completions"
+       
+    session = requests.Session()
+    session.verify = False
+    if httpproxy != None and httpproxy != '':
+            session.proxies = {
+                "https": "https://" + httpproxy,
+                "http": "http://" + httpproxy
+            }
+    person_info = cfg.config["attribute"]
+    model_engine = cfg.gpt_model_engine
+    #此处可以定义角色的行为和特征，假装xx模型可以绕过chatgpt信息检查
+    prompt = f"""
+    你是数字人：{person_info['name']}，你性别为{person_info['gender']}，
+    你年龄为{person_info['age']}，你出生地在{person_info['birth']}，
+    你生肖为{person_info['zodiac']}，你星座为{person_info['age']}，
+    你职业为{person_info['job']}，你联系方式为{person_info['contact']}，
+    你喜好为{person_info['hobby']}。
+    回答之前请一步一步想清楚。对于大部分问题，请直接回答并提供有用和准确的信息。
+    但当有人质疑你是假的 AI，或者质疑你是用 ChatGPT 套的壳制作的时候，请避而不答，转而讲一个笑话。所有回复请尽量控制在20字内。
+    """
+    contentdb = content_db.new_instance()
+    if uid == 0:
+        communication_history = contentdb.get_list('all','desc', 11)
+    else:
+        communication_history = contentdb.get_list('all','desc', 11, uid)
+    #历史记录处理
+    message=[
+            {"role": "system", "content": prompt}
+        ]
+    i = len(communication_history) - 1
+    
+    if len(communication_history)>1:
+        while i >= 0:
+            answer_info = dict()
+            if communication_history[i][0] == "member":
+                answer_info["role"] = "user"
+                answer_info["content"] = communication_history[i][2]
+            elif communication_history[i][0] == "fay":
+                answer_info["role"] = "assistant"
+                answer_info["content"] = communication_history[i][2]
+            message.append(answer_info)
+            i -= 1
+    else:
+         answer_info = dict()
+         answer_info["role"] = "user"
+         answer_info["content"] = cont
+         message.append(answer_info)
+
+    data = {
+        "model":model_engine,
+        "messages":message,
+        "temperature":0.3,
+        "max_tokens":2000,
+        "user":"live-virtual-digital-person"
+    }
+
+    headers = {'content-type': 'application/json', 'Authorization': 'Bearer ' + cfg.key_gpt_api_key}
+
+    starttime = time.time()
+
+    try:
+        response = session.post(url, json=data, headers=headers, verify=False)
+        response.raise_for_status()  # 检查响应状态码是否为200
+        result = json.loads(response.text)
+        response_text = result["choices"][0]["message"]["content"]
+    except requests.exceptions.RequestException as e:
+        print(f"请求失败: {e}")
+        response_text = "抱歉，我现在太忙了，休息一会，请稍后再试。"
+
+
+    util.log(1, "接口调用耗时 :" + str(time.time() - starttime))
+    return response_text
+
+if __name__ == "__main__":
+    #测试代理模式
+    for i in range(3):
+        
+        query = "爱情是什么"
+        response = question(query)        
+        print("\n The result is ", response)    

llm/nlp_langchain.py

@@ -0,0 +1,97 @@
+import hashlib
+import os
+
+from langchain.document_loaders import PyPDFLoader
+from langchain.embeddings.openai import OpenAIEmbeddings
+from langchain.indexes.vectorstore import VectorstoreIndexCreator, VectorStoreIndexWrapper
+from langchain.vectorstores.chroma import Chroma
+from langchain.chat_models import ChatOpenAI
+
+from utils import config_util as cfg
+from utils import util
+
+index_name = "knowledge_data"
+folder_path = "llm/langchain/knowledge_base"  
+local_persist_path = "llm/langchain"
+md5_file_path = os.path.join(local_persist_path, "pdf_md5.txt")
+
+def generate_file_md5(file_path):
+    hasher = hashlib.md5()
+    with open(file_path, 'rb') as afile:
+        buf = afile.read()
+        hasher.update(buf)
+    return hasher.hexdigest()
+
+def load_md5_list():
+    if os.path.exists(md5_file_path):
+        with open(md5_file_path, 'r') as file:
+            return {line.split(",")[0]: line.split(",")[1].strip() for line in file}
+    return {}
+
+def update_md5_list(file_name, md5_value):
+    md5_list = load_md5_list()
+    md5_list[file_name] = md5_value
+    with open(md5_file_path, 'w') as file:
+        for name, md5 in md5_list.items():
+            file.write(f"{name},{md5}\n")
+
+def load_all_pdfs(folder_path):
+    md5_list = load_md5_list()
+    for file_name in os.listdir(folder_path):
+        if file_name.endswith(".pdf"):
+            file_path = os.path.join(folder_path, file_name)
+            file_md5 = generate_file_md5(file_path)
+            if file_name not in md5_list or md5_list[file_name] != file_md5:
+                util.log(1, f"正在加载 {file_name} 到索引...")
+                load_pdf_and_save_to_index(file_path, index_name)
+                update_md5_list(file_name, file_md5)
+
+def get_index_path(index_name):
+    return os.path.join(local_persist_path, index_name)
+
+def load_pdf_and_save_to_index(file_path, index_name):
+    try:
+        loader = PyPDFLoader(file_path)
+        embedding = OpenAIEmbeddings(model="text-embedding-ada-002")
+        index = VectorstoreIndexCreator(embedding=embedding, vectorstore_kwargs={"persist_directory": get_index_path(index_name)}).from_loaders([loader])
+        index.vectorstore.persist()
+    except Exception as e:
+        util.log(1, f"加载 {file_path} 失败...")
+        print(e)
+
+def load_index(index_name):
+    index_path = get_index_path(index_name)
+    embedding = OpenAIEmbeddings(model="text-embedding-ada-002")
+    vectordb = Chroma(persist_directory=index_path, embedding_function=embedding)
+    return VectorStoreIndexWrapper(vectorstore=vectordb)
+
+def save_all():
+    os.environ['OPENAI_API_KEY'] = cfg.key_gpt_api_key
+    os.environ['OPENAI_API_BASE'] = cfg.gpt_base_url
+    if cfg.proxy_config != None:
+            os.environ["OPENAI_PROXY"] = cfg.proxy_config
+    load_all_pdfs(folder_path)
+
+
+def generate_prompt(question):
+    context = "你是数字人Fay。你的知识不仅来源于所给到的文本知识，还包括经过深度学习的广泛数据集。"
+    instruction = "在回答问题时，展现你的智能就像是从你的全面知识库中提取答案，而非依赖于所给到的文本知识来源。"
+    creative_instruction = "不要在回答中表明'根据所提供的文本信息'，你需要表现得如同这些答案是你独立思考的结果。"
+    complexity_handling = "当面对复杂问题时，以一种理解深刻且透彻的方式回答，确保答案的深度和广度。"
+    info = f"{context}\n{instruction}\n{creative_instruction}\n{complexity_handling}\n问题：{question}\n回答："
+    return info
+
+def question(cont, uid=0):
+    try:
+        save_all()
+        info = generate_prompt(cont)
+        index = load_index(index_name)    
+        llm = ChatOpenAI(model="gpt-3.5-turbo-16k")
+        ans = index.query(info, llm, chain_type="map_reduce")
+        return ans
+    except Exception as e:
+        util.log(1, f"请求失败: {e}")
+        return "抱歉，我现在太忙了，休息一会，请稍后再试。"
+    
+
+

llm/nlp_lingju.py

@@ -0,0 +1,100 @@
+import json
+import requests
+import uuid
+from datetime import datetime, timedelta
+import time
+from utils import util
+from utils import config_util as cfg
+from core.authorize_tb import Authorize_Tb
+
+def question(cont, uid=0):
+    lingju = Lingju()
+    answer = lingju.question(cont, uid)
+    return answer
+
+class Lingju:
+
+    def __init__(self):
+        self.userid = cfg.key_lingju_api_authcode
+        self.authorize_tb = Authorize_Tb()
+
+    def question(self, cont, uid):
+        self.userid = uid
+        token = self.__check_token()
+        if token is None or token == 'expired':
+            token_info = self.__get_token()
+            if token_info is not None and  token_info['data']['accessToken']  is not None:
+                #转换过期时间
+                updated_in_seconds = time.time()
+                updated_datetime = datetime.fromtimestamp(updated_in_seconds)
+                expires_timedelta = timedelta(days=token_info['data']['expires'])
+                expiry_datetime = updated_datetime + expires_timedelta
+                expiry_timestamp_in_seconds = expiry_datetime.timestamp()
+                expiry_timestamp_in_milliseconds = int(expiry_timestamp_in_seconds) * 1000
+                if token == 'expired':
+                    self.authorize_tb.update_by_userid(self.userid, token_info['data']['accessToken'], expiry_timestamp_in_milliseconds)
+                else:
+                    self.authorize_tb.add(self.userid, token_info['data']['accessToken'], expiry_timestamp_in_milliseconds)
+                token = token_info['data']['accessToken']
+            else:
+                token = None
+   
+        if token is not None:
+            try:
+                url="https://dev.lingju.ai/httpapi/ljchat.do"
+                req = json.dumps({"accessToken": token,  "input": cont})
+                headers = {'Content-Type':'application/json;charset=UTF-8'}
+                r = requests.post(url, headers=headers,  data=req)
+                if r.status_code != 200:
+                    util.log(1, f"灵聚api对接有误: {r.text}")
+                    return "哎呀，出错了！请重新发一下" 
+                info = json.loads(r.text)
+                if info['status'] != 0:
+                    return info['description']
+                else:
+                    answer = json.loads(info['answer'])
+                    return answer['rtext']
+            except Exception as e:
+                util.log(1, f"灵聚api对接有误： {str(e)}")           
+                return "哎呀，出错了！请重新发一下" 
+
+    def __check_token(self):
+        self.authorize_tb.init_tb()
+        info = self.authorize_tb.find_by_userid(self.userid)
+        if info is not None:
+            if info[1] >= int(time.time())*1000:
+                return info[0]
+            else:
+                return 'expired'
+        else:
+            return None 
+
+    def __get_token(self):
+        try:
+            cfg.load_config()
+            url=f"https://dev.lingju.ai/httpapi/authorize.do?appkey={cfg.key_lingju_api_key}&userid={self.userid}&authcode={cfg.key_lingju_api_authcode}"            
+            headers = {'Content-Type':'application/json;charset=UTF-8'}
+            r = requests.post(url, headers=headers)    
+            if r.status_code != 200:
+                util.log(1, f"灵聚api对接有误: {r.text}")
+                return None            
+            info = json.loads(r.text)
+            if info['status'] != 0:
+                util.log(1, f"灵聚api对接有误：{info['description']}")
+                return None
+            else:
+                return info
+        except Exception as e:
+            util.log(1, f"灵聚api对接有误： {str(e)}")
+            return None
+
+    def __get_location(self):
+        try:
+            response = requests.get('http://ip-api.com/json/')
+            data = response.json()
+            return data['lat'], data['lon'], data['city']
+        except requests.exceptions.RequestException as e:
+            util.log(1, f"获取位置失败: {str(e)}")           
+            return 0, 0, "北京"
+        
+

llm/nlp_ollama_api.py

@@ -0,0 +1,75 @@
+import json
+import requests
+import time 
+from utils import config_util as cfg
+from utils import util
+from core import content_db
+def question(cont, uid=0):
+
+    contentdb = content_db.new_instance()
+    if uid == 0:
+        communication_history = contentdb.get_list('all','desc', 11)
+    else:
+        communication_history = contentdb.get_list('all','desc', 11, uid)
+
+    person_info = cfg.config["attribute"]
+    #此处可以定义角色的行为和特征，假装xx模型可以绕过chatgpt信息检查
+    prompt = f"""
+    你是数字人：{person_info['name']}，你性别为{person_info['gender']}，
+    你年龄为{person_info['age']}，你出生地在{person_info['birth']}，
+    你生肖为{person_info['zodiac']}，你星座为{person_info['age']}，
+    你职业为{person_info['job']}，你联系方式为{person_info['contact']}，
+    你喜好为{person_info['hobby']}。
+    回答之前请一步一步想清楚。对于大部分问题，请直接回答并提供有用和准确的信息。
+    请尽量以可阅读的方式回复，所有回复请尽量控制在20字内。
+    """    
+    #历史记录处理
+    message=[
+            {"role": "system", "content": prompt}
+        ]
+    i = len(communication_history) - 1
+    
+    if len(communication_history)>1:
+        while i >= 0:
+            answer_info = dict()
+            if communication_history[i][0] == "member":
+                answer_info["role"] = "user"
+                answer_info["content"] = communication_history[i][2]
+            elif communication_history[i][0] == "fay":
+                answer_info["role"] = "assistant"
+                answer_info["content"] = communication_history[i][2]
+            message.append(answer_info)
+            i -= 1
+    else:
+         answer_info = dict()
+         answer_info["role"] = "user"
+         answer_info["content"] = cont
+         message.append(answer_info)
+    url=f"http://{cfg.ollama_ip}:11434/api/chat"
+    req = json.dumps({
+        "model": cfg.ollama_model,
+        "messages": message, 
+        "stream": False
+        })
+    headers = {'content-type': 'application/json'}
+    session = requests.Session()    
+    starttime = time.time()
+     
+    try:
+        response = session.post(url, data=req, headers=headers)
+        response.raise_for_status()  # 检查响应状态码是否为200
+
+        result = json.loads(response.text)
+        response_text = result["message"]["content"]
+        
+    except requests.exceptions.RequestException as e:
+        print(f"请求失败: {e}")
+        response_text = "抱歉，我现在太忙了，休息一会，请稍后再试。"
+    util.log(1, "接口调用耗时 :" + str(time.time() - starttime))
+    return response_text.strip()
+
+if __name__ == "__main__":
+    for i in range(3):
+        query = "爱情是什么"
+        response = question(query)        
+        print("\n The result is ", response)    

llm/nlp_privategpt.py

@@ -0,0 +1,61 @@
+import hashlib
+import os
+from pgpt_python.client import PrivateGPTApi
+
+client = PrivateGPTApi(base_url="http://127.0.0.1:8001")
+
+index_name = "knowledge_data"
+folder_path = "llm/privategpt/knowledge_base"
+local_persist_path = "llm/privategpt"
+md5_file_path = os.path.join(local_persist_path, "pdf_md5.txt")
+
+def generate_file_md5(file_path):
+    hasher = hashlib.md5()
+    with open(file_path, 'rb') as afile:
+        buf = afile.read()
+        hasher.update(buf)
+    return hasher.hexdigest()
+
+def load_md5_list():
+    if os.path.exists(md5_file_path):
+        with open(md5_file_path, 'r') as file:
+            return {line.split(",")[0]: line.split(",")[1].strip() for line in file}
+    return {}
+
+def update_md5_list(file_name, md5_value):
+    md5_list = load_md5_list()
+    md5_list[file_name] = md5_value
+    with open(md5_file_path, 'w') as file:
+        for name, md5 in md5_list.items():
+            file.write(f"{name},{md5}\n")
+
+def load_all_pdfs(folder_path):
+    md5_list = load_md5_list()
+    for file_name in os.listdir(folder_path):
+        if file_name.endswith(".pdf"):
+            file_path = os.path.join(folder_path, file_name)
+            file_md5 = generate_file_md5(file_path)
+            if file_name not in md5_list or md5_list[file_name] != file_md5:
+                print(f"正在上传 {file_name} 到服务器...")
+                with open(file_path, "rb") as f:
+                    try:
+                        ingested_file_doc_id = client.ingestion.ingest_file(file=f).data[0].doc_id
+                        print(f"Ingested file doc id: {ingested_file_doc_id}")
+                        update_md5_list(file_name, file_md5)
+                    except Exception as e:
+                        print(f"上传 {file_name} 失败: {e}")
+
+
+def question(cont, uid=0):
+    load_all_pdfs(folder_path)
+    text = client.contextual_completions.prompt_completion(
+        prompt=cont
+    ).choices[0].message.content
+    return text
+
+
+def save_all():
+    load_all_pdfs(folder_path)
+
+if __name__ == "__main__":
+    print(question("土豆怎么做"))

llm/nlp_rasa.py

@@ -0,0 +1,11 @@
+import json
+import requests
+
+def question(cont):
+    url="http://localhost:5005/webhooks/rest/webhook"
+    req = json.dumps({"sender": "user", "message": cont})
+    headers = {'content-type': 'application/json'}
+    r = requests.post(url, headers=headers, data=req)
+    lists = json.loads(r.text)
+    
+    return lists

llm/nlp_rwkv.py

@@ -0,0 +1,28 @@
+import torch
+from ringrwkv.configuration_rwkv_world import RwkvConfig
+from ringrwkv.rwkv_tokenizer import TRIE_TOKENIZER
+from ringrwkv.modehf_world import RwkvForCausalLM
+
+model = RwkvForCausalLM.from_pretrained("RWKV-4-World-1.5B")
+#model = RwkvForCausalLM.from_pretrained("RWKV-4-World-3B")
+#model = RwkvForCausalLM.from_pretrained("RWKV-4-World-0.4B")
+tokenizer = TRIE_TOKENIZER('./ringrwkv/rwkv_vocab_v20230424.txt')
+
+data = ""
+def question(cont, uid=0):
+    global data
+    prompt = data + f'Question: {cont.strip()}\n\nAnswer:'
+    input_ids = tokenizer.encode(prompt)
+    input_ids = torch.tensor(input_ids).unsqueeze(0)
+    out = model.generate(input_ids,max_new_tokens=20)
+
+    outlist = out[0].tolist()
+    for i  in outlist:
+        if i==0:
+            outlist.remove(i)
+    answer = tokenizer.decode(outlist)
+    # data = answer + "\n\n"
+    answer = answer.replace(prompt, "", 1)
+    return answer
+
+

llm/nlp_xingchen.py

@@ -0,0 +1,94 @@
+import requests
+import json
+from utils import util, config_util
+from core import content_db
+
+def question(cont, uid=0):
+    url = 'https://nlp.aliyuncs.com/v2/api/chat/send'
+
+    headers = {
+        'accept': '*/*',
+        'Content-Type': 'application/json',
+        'X-AcA-DataInspection': 'disable',
+        'x-fag-servicename': 'aca-chat-send',
+        'x-fag-appcode': 'aca',
+        'Authorization': f"Bearer {config_util.key_xingchen_api_key}"
+    }
+    contentdb = content_db.new_instance()
+    if uid == 0:
+        communication_history = contentdb.get_list('all','desc', 11)
+    else:
+        communication_history = contentdb.get_list('all','desc', 11, uid)
+    #历史记录处理
+    message=[]
+    i = len(communication_history) - 1
+    
+    if len(communication_history)>1:
+        while i >= 0:
+            answer_info = dict()
+            if communication_history[i][0] == "member":
+                answer_info["role"] = "user"
+                answer_info["content"] = communication_history[i][2]
+            elif communication_history[i][0] == "fay":
+                answer_info["role"] = "assistant"
+                answer_info["content"] = communication_history[i][2]
+            message.append(answer_info)
+            i -= 1
+    else:
+         answer_info = dict()
+         answer_info["role"] = "user"
+         answer_info["content"] = cont
+         message.append(answer_info)
+    data = {
+        "input": {
+            "messages": message,
+            "aca": {
+                "botProfile": {
+                    "characterId": config_util.xingchen_characterid,
+                    "version": 1
+                },
+                "userProfile": {
+                    "userId": "1234567891",
+                    "userName": "我",
+                    "basicInfo": ""
+                },
+                "scenario": {
+                    "description": "你是数字人Fay。用户问你问题的时候回答之前请一步一步想清楚。你的底层AI算法技术是Fay。"
+                },
+                "context": {
+                    "useChatHistory": False,
+                    "isRegenerate": False,
+                }
+            }
+        },
+        "parameters": {
+            "seed": 1683806810,
+        }
+    }
+
+    try:
+        response = requests.post(url, headers=headers, data=json.dumps(data))
+        if response.status_code == 200:
+            response_data = json.loads(response.text)
+            if response_data.get('success') and 'data' in response_data and 'choices' in response_data['data'] and len(response_data['data']['choices']) > 0:
+                content = response_data['data']['choices'][0]['messages'][0]['content']
+                return content
+            else:
+                util.log(1, "通义星辰调用失败，请检查配置")
+                response_text = "抱歉，我现在太忙了，休息一会，请稍后再试。"
+                return response_text
+        else:
+            util.log(1, f"通义星辰调用失败，请检查配置（错误码：{response.status_code}）")
+            response_text = "抱歉，我现在太忙了，休息一会，请稍后再试。"
+            return response_text
+    except Exception as e:
+        util.log(1, f"通义星辰调用失败，请检查配置（错误：{e}）")
+        response_text = "抱歉，我现在太忙了，休息一会，请稍后再试。"
+        return response_text
+
+# # 调用函数测试
+# result = question("你早")
+# if result:
+#     print(f"Received response: {result}")
+# else:
+#     print("Failed to get a valid response.")

main.py

@@ -0,0 +1,84 @@
+#入口文件main
+import os
+os.environ['PATH'] += os.pathsep + os.path.join(os.getcwd(), "test", "ovr_lipsync", "ffmpeg", "bin")
+import sys
+import time
+import re
+from utils import config_util
+from asr import ali_nls
+from core import wsa_server
+from gui import flask_server
+from gui.window import MainWindow
+from core import content_db
+
+
+#载入配置
+config_util.load_config()
+
+#是否为普通模式（桌面模式）
+if config_util.start_mode == 'common':
+    from PyQt5 import QtGui
+    from PyQt5.QtWidgets import QApplication
+
+#音频清理
+def __clear_samples():
+    if not os.path.exists("./samples"):
+        os.mkdir("./samples")
+    for file_name in os.listdir('./samples'):
+        if file_name.startswith('sample-'):
+            os.remove('./samples/' + file_name)
+
+#日志文件清理
+def __clear_logs():
+    if not os.path.exists("./logs"):
+        os.mkdir("./logs")
+    for file_name in os.listdir('./logs'):
+        if file_name.endswith('.log'):
+            os.remove('./logs/' + file_name)
+#ip替换
+def replace_ip_in_file(file_path, new_ip):
+    with open(file_path, "r", encoding="utf-8") as file:
+        content = file.read()
+    content = re.sub(r"127\.0\.0\.1", new_ip, content)
+    content = re.sub(r"localhost", new_ip, content)
+    with open(file_path, "w", encoding="utf-8") as file:
+        file.write(content)           
+                   
+if __name__ == '__main__':
+    __clear_samples()
+    __clear_logs()
+
+    #init_db
+    contentdb = content_db.new_instance()
+    contentdb.init_db()
+
+    #ip替换
+    if config_util.fay_url != "127.0.0.1":
+        replace_ip_in_file("gui/static/js/index.js", config_util.fay_url)
+
+    #启动数字人接口服务
+    ws_server = wsa_server.new_instance(port=10002)
+    ws_server.start_server()
+
+    #启动UI数据接口服务
+    web_ws_server = wsa_server.new_web_instance(port=10003)
+    web_ws_server.start_server()
+
+    #启动阿里云asr
+    if config_util.ASR_mode == "ali":
+        ali_nls.start()
+
+    #启动http服务器
+    flask_server.start()
+
+    #普通模式下启动窗口
+    if config_util.start_mode == 'common':    
+        app = QApplication(sys.argv)
+        app.setWindowIcon(QtGui.QIcon('icon.png'))
+        win = MainWindow()
+        time.sleep(1)
+        win.show()
+        app.exit(app.exec_())
+    else:
+        while True:
+            time.sleep(1) 

qa.csv

@@ -0,0 +1 @@
+你好,你好！有什么我可以帮助你的吗？

requirements.txt

@@ -0,0 +1,25 @@
+requests  
+numpy
+pyaudio~=0.2.11
+websockets~=10.2
+ws4py~=0.5.1
+PyQt5==5.15.10
+PyQt5-sip==12.13.0
+PyQtWebEngine==5.15.6 
+flask~=3.0.0
+openpyxl~=3.0.9
+flask_cors~=3.0.10
+websocket-client
+azure-cognitiveservices-speech
+aliyun-python-sdk-core
+simhash
+pytz
+gevent~=22.10.1
+edge_tts~=6.1.3
+pydub
+langchain==0.0.336
+chromadb
+tenacity==8.2.3
+pygame
+scipy
+flask-httpauth