Upload folder using huggingface_hub

README.md

@@ -1,52 +1,213 @@
----
-license: mit
----
+<div align="center">
 
+<h1>Retrieval-based-Voice-Conversion-WebUI</h1>
+一个基于VITS的简单易用的变声框架<br><br>
 
-# easyGUI
+[![madewithlove](https://img.shields.io/badge/made_with-%E2%9D%A4-red?style=for-the-badge&labelColor=orange
+)](https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI)
 
-`easyGUI` is a user-friendly voice conversion framework based on VITS, designed to eliminate timbre leakage by replacing input features with those from the training set. It's efficient even on lower-end GPUs, requiring only about 10 minutes of low-noise speech data for good results. The framework features a simple web interface, supports A card and I card acceleration, and uses the advanced RMVPE algorithm for pitch extraction.
+<img src="https://counter.seku.su/cmoe?name=rvc&theme=r34" /><br>
 
-## Installation
+[![Open In Colab](https://img.shields.io/badge/Colab-F9AB00?style=for-the-badge&logo=googlecolab&color=525252)](https://colab.research.google.com/github/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/blob/main/Retrieval_based_Voice_Conversion_WebUI.ipynb)
+[![Licence](https://img.shields.io/badge/LICENSE-MIT-green.svg?style=for-the-badge)](https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/blob/main/LICENSE)
+[![Huggingface](https://img.shields.io/badge/🤗%20-Spaces-yellow.svg?style=for-the-badge)](https://huggingface.co/lj1995/VoiceConversionWebUI/tree/main/)
 
-### Prerequisites
-- Python 3.8 or higher
+[![Discord](https://img.shields.io/badge/RVC%20Developers-Discord-7289DA?style=for-the-badge&logo=discord&logoColor=white)](https://discord.gg/HcsmBBGyVk)
 
-### Installation Steps
-1. **Install Pytorch**:
-   ```bash
-   pip install torch torchvision torchaudio
-   ```
-   
+[**更新日志**](https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/blob/main/docs/Changelog_CN.md) | [**常见问题解答**](https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/wiki/%E5%B8%B8%E8%A7%81%E9%97%AE%E9%A2%98%E8%A7%A3%E7%AD%94) | [**AutoDL·5毛钱训练AI歌手**](https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/wiki/Autodl%E8%AE%AD%E7%BB%83RVC%C2%B7AI%E6%AD%8C%E6%89%8B%E6%95%99%E7%A8%8B) | [**对照实验记录**](https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/wiki/Autodl%E8%AE%AD%E7%BB%83RVC%C2%B7AI%E6%AD%8C%E6%89%8B%E6%95%99%E7%A8%8B](https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/wiki/%E5%AF%B9%E7%85%A7%E5%AE%9E%E9%AA%8C%C2%B7%E5%AE%9E%E9%AA%8C%E8%AE%B0%E5%BD%95)) | [**在线演示**](https://modelscope.cn/studios/FlowerCry/RVCv2demo)
 
-2. **Install Dependencies**:
-     ```bash
-     pip install -r requirements.txt
-     ```
-   
+[**English**](./docs/en/README.en.md) | [**中文简体**](./README.md) | [**日本語**](./docs/jp/README.ja.md) | [**한국어**](./docs/kr/README.ko.md) ([**韓國語**](./docs/kr/README.ko.han.md)) | [**Français**](./docs/fr/README.fr.md) | [**Türkçe**](./docs/tr/README.tr.md) | [**Português**](./docs/pt/README.pt.md)
 
-3
+</div>
 
-### Additional Setup
-- **Download Assets**:
-  Download necessary models and files using the scripts in the `tools` directory.
-- **Install FFmpeg**:
-  ```bash
-  sudo apt install ffmpeg  
-  ```
+> 底模使用接近50小时的开源高质量VCTK训练集训练，无版权方面的顾虑，请大家放心使用
 
-## Usage
-Start the WebUI:
+> 请期待RVCv3的底模，参数更大，数据更大，效果更好，基本持平的推理速度，需要训练数据量更少。
+
+<table>
+   <tr>
+		<td align="center">训练推理界面</td>
+		<td align="center">实时变声界面</td>
+	</tr>
+  <tr>
+		<td align="center"><img src="https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/assets/129054828/092e5c12-0d49-4168-a590-0b0ef6a4f630"></td>
+    <td align="center"><img src="https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/assets/129054828/730b4114-8805-44a1-ab1a-04668f3c30a6"></td>
+	</tr>
+	<tr>
+		<td align="center">go-web.bat</td>
+		<td align="center">go-realtime-gui.bat</td>
+	</tr>
+  <tr>
+    <td align="center">可以自由选择想要执行的操作。</td>
+		<td align="center">我们已经实现端到端170ms延迟。如使用ASIO输入输出设备，已能实现端到端90ms延迟，但非常依赖硬件驱动支持。</td>
+	</tr>
+</table>
+
+## 简介
+本仓库具有以下特点
++ 使用top1检索替换输入源特征为训练集特征来杜绝音色泄漏
++ 即便在相对较差的显卡上也能快速训练
++ 使用少量数据进行训练也能得到较好结果(推荐至少收集10分钟低底噪语音数据)
++ 可以通过模型融合来改变音色(借助ckpt处理选项卡中的ckpt-merge)
++ 简单易用的网页界面
++ 可调用UVR5模型来快速分离人声和伴奏
++ 使用最先进的[人声音高提取算法InterSpeech2023-RMVPE](#参考项目)根绝哑音问题。效果最好（显著地）但比crepe_full更快、资源占用更小
++ A卡I卡加速支持
+
+点此查看我们的[演示��频](https://www.bilibili.com/video/BV1pm4y1z7Gm/) !
+
+## 环境配置
+以下指令需在 Python 版本大于3.8的环境中执行。  
+
+### Windows/Linux/MacOS等平台通用方法
+下列方法任选其一。
+#### 1. 通过 pip 安装依赖
+1. 安装Pytorch及其核心依赖，若已安装则跳过。参考自: https://pytorch.org/get-started/locally/
+```bash
+pip install torch torchvision torchaudio
+```
+2. 如果是 win 系统 + Nvidia Ampere 架构(RTX30xx)，根据 #21 的经验，需要指定 pytorch 对应的 cuda 版本
+```bash
+pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117
+```
+3. 根据自己的显卡安装对应依赖
+- N卡
+```bash
+pip install -r requirements.txt
+```
+- A卡/I卡
+```bash
+pip install -r requirements-dml.txt
+```
+- A卡ROCM(Linux)
+```bash
+pip install -r requirements-amd.txt
+```
+- I卡IPEX(Linux)
+```bash
+pip install -r requirements-ipex.txt
+```
+
+#### 2. 通过 poetry 来安装依赖
+安装 Poetry 依赖管理工具，若已安装则跳过。参考自: https://python-poetry.org/docs/#installation
+```bash
+curl -sSL https://install.python-poetry.org | python3 -
+```
+
+通过 Poetry 安装依赖时，python 建议使用 3.7-3.10 版本，其余版本在安装 llvmlite==0.39.0 时会出现冲突
+```bash
+poetry init -n
+poetry env use "path to your python.exe"
+poetry run pip install -r requirments.txt
+```
+
+### MacOS
+可以通过 `run.sh` 来安装依赖
+```bash
+sh ./run.sh
+```
+
+## 其他预模型准备
+RVC需要其他一些预模型来推理和训练。
+
+你可以从我们的[Hugging Face space](https://huggingface.co/lj1995/VoiceConversionWebUI/tree/main/)下载到这些模型。
+
+### 1. 下载 assets
+以下是一份清单，包括了所有RVC所需的预模型和其他文件的名称。你可以在`tools`文件夹找到下载它们的脚本。
+
+- ./assets/hubert/hubert_base.pt
+
+- ./assets/pretrained 
+
+- ./assets/uvr5_weights
+
+想使用v2版本模型的话，需要额外下载
+
+- ./assets/pretrained_v2
+
+### 2. 安装 ffmpeg
+若ffmpeg和ffprobe已安装则跳过。
+
+#### Ubuntu/Debian 用户
+```bash
+sudo apt install ffmpeg
+```
+#### MacOS 用户
+```bash
+brew install ffmpeg
+```
+#### Windows 用户
+下载后放置在根目录。
+- 下载[ffmpeg.exe](https://huggingface.co/lj1995/VoiceConversionWebUI/blob/main/ffmpeg.exe)
+
+- 下载[ffprobe.exe](https://huggingface.co/lj1995/VoiceConversionWebUI/blob/main/ffprobe.exe)
+
+### 3. 下载 rmvpe 人声音高提取算法所需文件
+
+如果你想使用最新的RMVPE人声音高提取算法，则你需要下载音高提取模型参数并放置于RVC根目录。
+
+- 下载[rmvpe.pt](https://huggingface.co/lj1995/VoiceConversionWebUI/blob/main/rmvpe.pt)
+
+#### 下载 rmvpe 的 dml 环境(可选, A卡/I卡用户)
+
+- 下载[rmvpe.onnx](https://huggingface.co/lj1995/VoiceConversionWebUI/blob/main/rmvpe.onnx)
+
+### 4. AMD显卡Rocm(可选, 仅Linux)
+
+如果你想基于AMD的Rocm技术在Linux系统上运行RVC，请先在[这里](https://rocm.docs.amd.com/en/latest/deploy/linux/os-native/install.html)安装所需的驱动。
+
+若你使用的是Arch Linux，可以使用pacman来安装所需驱动：
+````
+pacman -S rocm-hip-sdk rocm-opencl-sdk
+````
+对于某些型号的显卡，你可能需要额外配置如下的环境变量（如：RX6700XT）：
+````
+export ROCM_PATH=/opt/rocm
+export HSA_OVERRIDE_GFX_VERSION=10.3.0
+````
+同时确保你的当前用户处于`render`与`video`用户组内：
+````
+sudo usermod -aG render $USERNAME
+sudo usermod -aG video $USERNAME
+````
+
+## 开始使用
+### 直接启动
+使用以下指令来启动 WebUI
 ```bash
-python demo.py
+python infer-web.py
 ```
 
+若先前使用 Poetry 安装依赖，则可以通过以下方式启动WebUI
+```bash
+poetry run python infer-web.py
+```
 
-## Features
-- Top1 retrieval to replace input features
-- Fast training on less powerful GPUs
-- Model merging to change timbre
-- Advanced pitch extraction with RMVPE
+### 使用整合包
+下载并解压`RVC-beta.7z`
+#### Windows 用户
+双击`go-web.bat`
+#### MacOS 用户
+```bash
+sh ./run.sh
+```
+### 对于需要使用IPEX技术的I卡用户(仅Linux)
+```bash
+source /opt/intel/oneapi/setvars.sh
+```
 
+## 参考项目
++ [ContentVec](https://github.com/auspicious3000/contentvec/)
++ [VITS](https://github.com/jaywalnut310/vits)
++ [HIFIGAN](https://github.com/jik876/hifi-gan)
++ [Gradio](https://github.com/gradio-app/gradio)
++ [FFmpeg](https://github.com/FFmpeg/FFmpeg)
++ [Ultimate Vocal Remover](https://github.com/Anjok07/ultimatevocalremovergui)
++ [audio-slicer](https://github.com/openvpi/audio-slicer)
++ [Vocal pitch extraction:RMVPE](https://github.com/Dream-High/RMVPE)
+  + The pretrained model is trained and tested by [yxlllc](https://github.com/yxlllc/RMVPE) and [RVC-Boss](https://github.com/RVC-Boss).
 
----
+## 感谢所有贡献者作出的努力
+<a href="https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/graphs/contributors" target="_blank">
+  <img src="https://contrib.rocks/image?repo=RVC-Project/Retrieval-based-Voice-Conversion-WebUI" />
+</a>

infer/modules/vc/__init__.py

@@ -1,5 +0,0 @@
-from .pipeline import Pipeline
-from .modules import VC
-from .utils import get_index_path_from_model, load_hubert
-from .info import show_info
-from .hash import model_hash_ckpt, hash_id, hash_similarity

infer/modules/vc/modules.py

@@ -1,6 +1,5 @@
 import traceback
 import logging
-import os
 
 logger = logging.getLogger(__name__)
 
@@ -10,10 +9,14 @@ import torch
 from io import BytesIO
 
 from infer.lib.audio import load_audio, wav2
-from rvc.synthesizer import get_synthesizer, load_synthesizer
-from .info import show_model_info
-from .pipeline import Pipeline
-from .utils import get_index_path_from_model, load_hubert
+from infer.lib.infer_pack.models import (
+    SynthesizerTrnMs256NSFsid,
+    SynthesizerTrnMs256NSFsid_nono,
+    SynthesizerTrnMs768NSFsid,
+    SynthesizerTrnMs768NSFsid_nono,
+)
+from infer.modules.vc.pipeline import Pipeline
+from infer.modules.vc.utils import *
 
 
 class VC:
@@ -59,45 +62,71 @@ class VC:
                 ) = None
                 if torch.cuda.is_available():
                     torch.cuda.empty_cache()
-                elif torch.backends.mps.is_available():
-                    torch.mps.empty_cache()
                 ###楼下不这么折腾清理不干净
-                self.net_g, self.cpt = get_synthesizer(self.cpt, self.config.device)
                 self.if_f0 = self.cpt.get("f0", 1)
                 self.version = self.cpt.get("version", "v1")
+                if self.version == "v1":
+                    if self.if_f0 == 1:
+                        self.net_g = SynthesizerTrnMs256NSFsid(
+                            *self.cpt["config"], is_half=self.config.is_half
+                        )
+                    else:
+                        self.net_g = SynthesizerTrnMs256NSFsid_nono(*self.cpt["config"])
+                elif self.version == "v2":
+                    if self.if_f0 == 1:
+                        self.net_g = SynthesizerTrnMs768NSFsid(
+                            *self.cpt["config"], is_half=self.config.is_half
+                        )
+                    else:
+                        self.net_g = SynthesizerTrnMs768NSFsid_nono(*self.cpt["config"])
                 del self.net_g, self.cpt
                 if torch.cuda.is_available():
                     torch.cuda.empty_cache()
-                elif torch.backends.mps.is_available():
-                    torch.mps.empty_cache()
             return (
-                (
-                    {"visible": False, "__type__": "update"},
-                    to_return_protect0,
-                    to_return_protect1,
-                    {"value": to_return_protect[2], "__type__": "update"},
-                    {"value": to_return_protect[3], "__type__": "update"},
-                    {"value": "", "__type__": "update"},
-                )
-                if to_return_protect
-                else {"visible": True, "maximum": 0, "__type__": "update"}
+                {"visible": False, "__type__": "update"},
+                {
+                    "visible": True,
+                    "value": to_return_protect0,
+                    "__type__": "update",
+                },
+                {
+                    "visible": True,
+                    "value": to_return_protect1,
+                    "__type__": "update",
+                },
+                "",
+                "",
             )
-
         person = f'{os.getenv("weight_root")}/{sid}'
         logger.info(f"Loading: {person}")
 
-        self.net_g, self.cpt = load_synthesizer(person, self.config.device)
+        self.cpt = torch.load(person, map_location="cpu")
         self.tgt_sr = self.cpt["config"][-1]
         self.cpt["config"][-3] = self.cpt["weight"]["emb_g.weight"].shape[0]  # n_spk
         self.if_f0 = self.cpt.get("f0", 1)
         self.version = self.cpt.get("version", "v1")
 
+        synthesizer_class = {
+            ("v1", 1): SynthesizerTrnMs256NSFsid,
+            ("v1", 0): SynthesizerTrnMs256NSFsid_nono,
+            ("v2", 1): SynthesizerTrnMs768NSFsid,
+            ("v2", 0): SynthesizerTrnMs768NSFsid_nono,
+        }
+
+        self.net_g = synthesizer_class.get(
+            (self.version, self.if_f0), SynthesizerTrnMs256NSFsid
+        )(*self.cpt["config"], is_half=self.config.is_half)
+
+        del self.net_g.enc_q
+
+        self.net_g.load_state_dict(self.cpt["weight"], strict=False)
+        self.net_g.eval().to(self.config.device)
         if self.config.is_half:
             self.net_g = self.net_g.half()
         else:
             self.net_g = self.net_g.float()
-        self.pipeline = Pipeline(self.tgt_sr, self.config)
 
+        self.pipeline = Pipeline(self.tgt_sr, self.config)
         n_spk = self.cpt["config"][-3]
         index = {"value": get_index_path_from_model(sid), "__type__": "update"}
         logger.info("Select index: " + index["value"])
@@ -109,7 +138,6 @@ class VC:
                 to_return_protect1,
                 index,
                 index,
-                show_model_info(self.cpt),
             )
             if to_return_protect
             else {"visible": True, "maximum": n_spk, "__type__": "update"}
@@ -132,22 +160,18 @@ class VC:
     ):
         if input_audio_path is None:
             return "You need to upload an audio", None
-        elif hasattr(input_audio_path, "name"):
-            input_audio_path = str(input_audio_path.name)
         f0_up_key = int(f0_up_key)
         try:
             audio = load_audio(input_audio_path, 16000)
             audio_max = np.abs(audio).max() / 0.95
             if audio_max > 1:
-                np.divide(audio, audio_max, audio)
+                audio /= audio_max
             times = [0, 0, 0]
 
             if self.hubert_model is None:
-                self.hubert_model = load_hubert(self.config.device, self.config.is_half)
+                self.hubert_model = load_hubert(self.config)
 
             if file_index:
-                if hasattr(file_index, "name"):
-                    file_index = str(file_index.name)
                 file_index = (
                     file_index.strip(" ")
                     .strip('"')
@@ -166,6 +190,7 @@ class VC:
                 self.net_g,
                 sid,
                 audio,
+                input_audio_path,
                 times,
                 f0_up_key,
                 f0_method,
@@ -179,25 +204,25 @@ class VC:
                 self.version,
                 protect,
                 f0_file,
-            ).astype(np.int16)
+            )
             if self.tgt_sr != resample_sr >= 16000:
                 tgt_sr = resample_sr
             else:
                 tgt_sr = self.tgt_sr
             index_info = (
-                "Index: %s." % file_index
+                "Index:\n%s." % file_index
                 if os.path.exists(file_index)
                 else "Index not used."
             )
             return (
-                "Success.\n%s\nTime: npy: %.2fs, f0: %.2fs, infer: %.2fs."
+                "Success.\n%s\nTime:\nnpy: %.2fs, f0: %.2fs, infer: %.2fs."
                 % (index_info, *times),
                 (tgt_sr, audio_opt),
             )
-        except Exception as e:
+        except:
             info = traceback.format_exc()
             logger.warning(info)
-            return str(e), None
+            return info, (None, None)
 
     def vc_multi(
         self,

infer/modules/vc/pipeline.py

@@ -5,22 +5,40 @@ import logging
 
 logger = logging.getLogger(__name__)
 
-from time import time
+from functools import lru_cache
+from time import time as ttime
 
 import faiss
 import librosa
 import numpy as np
+import parselmouth
+import pyworld
 import torch
 import torch.nn.functional as F
+import torchcrepe
 from scipy import signal
 
-from rvc.f0 import PM, Harvest, RMVPE, CRePE, Dio, FCPE
-
 now_dir = os.getcwd()
 sys.path.append(now_dir)
 
 bh, ah = signal.butter(N=5, Wn=48, btype="high", fs=16000)
 
+input_audio_path2wav = {}
+
+
+@lru_cache
+def cache_harvest_f0(input_audio_path, fs, f0max, f0min, frame_period):
+    audio = input_audio_path2wav[input_audio_path]
+    f0, t = pyworld.harvest(
+        audio,
+        fs=fs,
+        f0_ceil=f0max,
+        f0_floor=f0min,
+        frame_period=frame_period,
+    )
+    f0 = pyworld.stonemask(audio, f0, t, fs)
+    return f0
+
 
 def change_rms(data1, sr1, data2, sr2, rate):  # 1是输入音频，2是输出音频,rate是2的占比
     # print(data1.max(),data2.max())
@@ -65,6 +83,7 @@ class Pipeline(object):
 
     def get_f0(
         self,
+        input_audio_path,
         x,
         p_len,
         f0_up_key,
@@ -72,62 +91,73 @@ class Pipeline(object):
         filter_radius,
         inp_f0=None,
     ):
+        global input_audio_path2wav
+        time_step = self.window / self.sr * 1000
         f0_min = 50
         f0_max = 1100
         f0_mel_min = 1127 * np.log(1 + f0_min / 700)
         f0_mel_max = 1127 * np.log(1 + f0_max / 700)
         if f0_method == "pm":
-            if not hasattr(self, "pm"):
-                self.pm = PM(self.window, f0_min, f0_max, self.sr)
-            f0 = self.pm.compute_f0(x, p_len=p_len)
-        if f0_method == "dio":
-            if not hasattr(self, "dio"):
-                self.dio = Dio(self.window, f0_min, f0_max, self.sr)
-            f0 = self.dio.compute_f0(x, p_len=p_len)
+            f0 = (
+                parselmouth.Sound(x, self.sr)
+                .to_pitch_ac(
+                    time_step=time_step / 1000,
+                    voicing_threshold=0.6,
+                    pitch_floor=f0_min,
+                    pitch_ceiling=f0_max,
+                )
+                .selected_array["frequency"]
+            )
+            pad_size = (p_len - len(f0) + 1) // 2
+            if pad_size > 0 or p_len - len(f0) - pad_size > 0:
+                f0 = np.pad(
+                    f0, [[pad_size, p_len - len(f0) - pad_size]], mode="constant"
+                )
         elif f0_method == "harvest":
-            if not hasattr(self, "harvest"):
-                self.harvest = Harvest(self.window, f0_min, f0_max, self.sr)
-            f0 = self.harvest.compute_f0(x, p_len=p_len, filter_radius=filter_radius)
+            input_audio_path2wav[input_audio_path] = x.astype(np.double)
+            f0 = cache_harvest_f0(input_audio_path, self.sr, f0_max, f0_min, 10)
+            if filter_radius > 2:
+                f0 = signal.medfilt(f0, 3)
         elif f0_method == "crepe":
-            if not hasattr(self, "crepe"):
-                self.crepe = CRePE(
-                    self.window,
-                    f0_min,
-                    f0_max,
-                    self.sr,
-                    self.device,
-                )
-            f0 = self.crepe.compute_f0(x, p_len=p_len)
+            model = "full"
+            # Pick a batch size that doesn't cause memory errors on your gpu
+            batch_size = 512
+            # Compute pitch using first gpu
+            audio = torch.tensor(np.copy(x))[None].float()
+            f0, pd = torchcrepe.predict(
+                audio,
+                self.sr,
+                self.window,
+                f0_min,
+                f0_max,
+                model,
+                batch_size=batch_size,
+                device=self.device,
+                return_periodicity=True,
+            )
+            pd = torchcrepe.filter.median(pd, 3)
+            f0 = torchcrepe.filter.mean(f0, 3)
+            f0[pd < 0.1] = 0
+            f0 = f0[0].cpu().numpy()
         elif f0_method == "rmvpe":
-            if not hasattr(self, "rmvpe"):
+            if not hasattr(self, "model_rmvpe"):
+                from infer.lib.rmvpe import RMVPE
+
                 logger.info(
-                    "Loading rmvpe model %s" % "%s/rmvpe.pt" % os.environ["rmvpe_root"]
+                    "Loading rmvpe model,%s" % "%s/rmvpe.pt" % os.environ["rmvpe_root"]
                 )
-                self.rmvpe = RMVPE(
+                self.model_rmvpe = RMVPE(
                     "%s/rmvpe.pt" % os.environ["rmvpe_root"],
                     is_half=self.is_half,
                     device=self.device,
-                    # use_jit=self.config.use_jit,
                 )
-            f0 = self.rmvpe.compute_f0(x, p_len=p_len, filter_radius=0.03)
+            f0 = self.model_rmvpe.infer_from_audio(x, thred=0.03)
 
             if "privateuseone" in str(self.device):  # clean ortruntime memory
-                del self.rmvpe.model
-                del self.rmvpe
+                del self.model_rmvpe.model
+                del self.model_rmvpe
                 logger.info("Cleaning ortruntime memory")
 
-        elif f0_method == "fcpe":
-            if not hasattr(self, "model_fcpe"):
-                logger.info("Loading fcpe model")
-                self.model_fcpe = FCPE(
-                    self.window,
-                    f0_min,
-                    f0_max,
-                    self.sr,
-                    self.device,
-                )
-            f0 = self.model_fcpe.compute_f0(x, p_len=p_len)
-
         f0 *= pow(2, f0_up_key / 12)
         # with open("test.txt","w")as f:f.write("\n".join([str(i)for i in f0.tolist()]))
         tf0 = self.sr // self.window  # 每秒f0点数
@@ -184,7 +214,7 @@ class Pipeline(object):
             "padding_mask": padding_mask,
             "output_layer": 9 if version == "v1" else 12,
         }
-        t0 = time()
+        t0 = ttime()
         with torch.no_grad():
             logits = model.extract_features(**inputs)
             feats = model.final_proj(logits[0]) if version == "v1" else logits[0]
@@ -202,10 +232,7 @@ class Pipeline(object):
             # _, I = index.search(npy, 1)
             # npy = big_npy[I.squeeze()]
 
-            try:
-                score, ix = index.search(npy, k=8)
-            except:
-                raise Exception("index mistatch")
+            score, ix = index.search(npy, k=8)
             weight = np.square(1 / score)
             weight /= weight.sum(axis=1, keepdims=True)
             npy = np.sum(big_npy[ix] * np.expand_dims(weight, axis=2), axis=1)
@@ -222,7 +249,7 @@ class Pipeline(object):
             feats0 = F.interpolate(feats0.permute(0, 2, 1), scale_factor=2).permute(
                 0, 2, 1
             )
-        t1 = time()
+        t1 = ttime()
         p_len = audio0.shape[0] // self.window
         if feats.shape[1] < p_len:
             p_len = feats.shape[1]
@@ -239,26 +266,14 @@ class Pipeline(object):
             feats = feats.to(feats0.dtype)
         p_len = torch.tensor([p_len], device=self.device).long()
         with torch.no_grad():
-            audio1 = (
-                (
-                    net_g.infer(
-                        feats,
-                        p_len,
-                        sid,
-                        pitch=pitch,
-                        pitchf=pitchf,
-                    )[0, 0]
-                )
-                .data.cpu()
-                .float()
-                .numpy()
-            )
+            hasp = pitch is not None and pitchf is not None
+            arg = (feats, p_len, pitch, pitchf, sid) if hasp else (feats, p_len, sid)
+            audio1 = (net_g.infer(*arg)[0][0, 0]).data.cpu().float().numpy()
+            del hasp, arg
         del feats, p_len, padding_mask
         if torch.cuda.is_available():
             torch.cuda.empty_cache()
-        elif torch.backends.mps.is_available():
-            torch.mps.empty_cache()
-        t2 = time()
+        t2 = ttime()
         times[0] += t1 - t0
         times[2] += t2 - t1
         return audio1
@@ -269,6 +284,7 @@ class Pipeline(object):
         net_g,
         sid,
         audio,
+        input_audio_path,
         times,
         f0_up_key,
         f0_method,
@@ -292,6 +308,7 @@ class Pipeline(object):
         ):
             try:
                 index = faiss.read_index(file_index)
+                # big_npy = np.load(file_big_npy)
                 big_npy = index.reconstruct_n(0, index.ntotal)
             except:
                 traceback.print_exc()
@@ -317,7 +334,7 @@ class Pipeline(object):
         s = 0
         audio_opt = []
         t = None
-        t1 = time()
+        t1 = ttime()
         audio_pad = np.pad(audio, (self.t_pad, self.t_pad), mode="reflect")
         p_len = audio_pad.shape[0] // self.window
         inp_f0 = None
@@ -333,29 +350,27 @@ class Pipeline(object):
                 traceback.print_exc()
         sid = torch.tensor(sid, device=self.device).unsqueeze(0).long()
         pitch, pitchf = None, None
-        if if_f0:
-            if if_f0 == 1:
-                pitch, pitchf = self.get_f0(
-                    audio_pad,
-                    p_len,
-                    f0_up_key,
-                    f0_method,
-                    filter_radius,
-                    inp_f0,
-                )
-            elif if_f0 == 2:
-                pitch, pitchf = f0_method
+        if if_f0 == 1:
+            pitch, pitchf = self.get_f0(
+                input_audio_path,
+                audio_pad,
+                p_len,
+                f0_up_key,
+                f0_method,
+                filter_radius,
+                inp_f0,
+            )
             pitch = pitch[:p_len]
             pitchf = pitchf[:p_len]
             if "mps" not in str(self.device) or "xpu" not in str(self.device):
                 pitchf = pitchf.astype(np.float32)
             pitch = torch.tensor(pitch, device=self.device).unsqueeze(0).long()
             pitchf = torch.tensor(pitchf, device=self.device).unsqueeze(0).float()
-        t2 = time()
+        t2 = ttime()
         times[1] += t2 - t1
         for t in opt_ts:
             t = t // self.window * self.window
-            if if_f0:
+            if if_f0 == 1:
                 audio_opt.append(
                     self.vc(
                         model,
@@ -390,7 +405,7 @@ class Pipeline(object):
                     )[self.t_pad_tgt : -self.t_pad_tgt]
                 )
             s = t
-        if if_f0:
+        if if_f0 == 1:
             audio_opt.append(
                 self.vc(
                     model,
@@ -435,10 +450,8 @@ class Pipeline(object):
         max_int16 = 32768
         if audio_max > 1:
             max_int16 /= audio_max
-        np.multiply(audio_opt, max_int16, audio_opt)
+        audio_opt = (audio_opt * max_int16).astype(np.int16)
         del pitch, pitchf, sid
         if torch.cuda.is_available():
             torch.cuda.empty_cache()
-        elif torch.backends.mps.is_available():
-            torch.mps.empty_cache()
         return audio_opt

infer/modules/vc/utils.py

@@ -9,8 +9,7 @@ def get_index_path_from_model(sid):
             f
             for f in [
                 os.path.join(root, name)
-                for path in [os.getenv("outside_index_root"), os.getenv("index_root")]
-                for root, _, files in os.walk(path, topdown=False)
+                for root, _, files in os.walk(os.getenv("index_root"), topdown=False)
                 for name in files
                 if name.endswith(".index") and "trained" not in name
             ]
@@ -20,14 +19,14 @@ def get_index_path_from_model(sid):
     )
 
 
-def load_hubert(device, is_half):
+def load_hubert(config):
     models, _, _ = checkpoint_utils.load_model_ensemble_and_task(
         ["assets/hubert/hubert_base.pt"],
         suffix="",
     )
     hubert_model = models[0]
-    hubert_model = hubert_model.to(device)
-    if is_half:
+    hubert_model = hubert_model.to(config.device)
+    if config.is_half:
         hubert_model = hubert_model.half()
     else:
         hubert_model = hubert_model.float()

requirements.txt

@@ -1,15 +1,16 @@
 joblib>=1.1.0
-numba
+numba==0.56.4
 numpy==1.23.5
 scipy
 librosa==0.9.1
-llvmlite
-fairseq
-faiss-cpu
-gradio
+llvmlite==0.39.0
+fairseq==0.12.2
+faiss-cpu==1.7.3
+gradio==3.34.0
 Cython
 pydub>=0.25.1
 soundfile>=0.12.1
+ffmpeg-python>=0.2.0
 tensorboardX
 Jinja2>=3.1.2
 json5
@@ -40,8 +41,8 @@ httpx
 onnxruntime; sys_platform == 'darwin'
 onnxruntime-gpu; sys_platform != 'darwin'
 torchcrepe==0.0.20
-fastapi
+fastapi==0.88
 torchfcpe
+ffmpy==0.3.1
 python-dotenv>=1.0.0
 av
-pybase16384