Whisper-AI - XWiki

= Whisper-WebUI =

A Gradio-based browser interface for [[Whisper>>url:https://github.com/openai/whisper]]. You can use it as an Easy Subtitle Generator!

5

6

[[~[~[image:https://private-user-images.githubusercontent.com/97279763/396437782-caea3afd-a73c-40af-a347-8d57914b1d0f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDEzMDA1MTQsIm5iZiI6MTc0MTMwMDIxNCwicGF0aCI6Ii85NzI3OTc2My8zOTY0Mzc3ODItY2FlYTNhZmQtYTczYy00MGFmLWEzNDctOGQ1NzkxNGIxZDBmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAzMDYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMzA2VDIyMzAxNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTY2MjEzMGY2YWEyMzM4ODAzZTk3YmFiY2EyNWMzOGIwMjdhM2JhMmZiNDZjZTQzNDY5MDFiODBiYmMwNDk2NWImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.uBtCRmRlYf3zF2aULRd--eajIi4EtuumFyaFSjH5O4s~|~|alt="screen"~]~]>>url:https://private-user-images.githubusercontent.com/97279763/396437782-caea3afd-a73c-40af-a347-8d57914b1d0f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDEzMDA1MTQsIm5iZiI6MTc0MTMwMDIxNCwicGF0aCI6Ii85NzI3OTc2My8zOTY0Mzc3ODItY2FlYTNhZmQtYTczYy00MGFmLWEzNDctOGQ1NzkxNGIxZDBmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAzMDYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMzA2VDIyMzAxNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTY2MjEzMGY2YWEyMzM4ODAzZTk3YmFiY2EyNWMzOGIwMjdhM2JhMmZiNDZjZTQzNDY5MDFiODBiYmMwNDk2NWImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.uBtCRmRlYf3zF2aULRd--eajIi4EtuumFyaFSjH5O4s]]

== Notebook ==

If you wish to try this on Colab, you can do it in [[here>>url:https://colab.research.google.com/github/jhj0517/Whisper-WebUI/blob/master/notebook/whisper-webui.ipynb]]!

= Feature =

* Select the Whisper implementation you want to use between :

17

** [[openai/whisper>>url:https://github.com/openai/whisper]]

18

** [[SYSTRAN/faster-whisper>>url:https://github.com/SYSTRAN/faster-whisper]] (used by default)

19

** [[Vaibhavs10/insanely-fast-whisper>>url:https://github.com/Vaibhavs10/insanely-fast-whisper]]

20

* Generate subtitles from various sources, including :

** Files

** Youtube

** Microphone

* Currently supported subtitle formats :

25

** SRT

26

** WebVTT

27

** txt ( only text file without timeline )

28

* Speech to Text Translation

29

** From other languages to English. ( This is Whisper's end-to-end speech-to-text translation feature )

30

* Text to Text Translation

31

** Translate subtitle files using Facebook NLLB models

32

** Translate subtitle files using DeepL API

33

* Pre-processing audio input with [[Silero VAD>>url:https://github.com/snakers4/silero-vad]].

34

* Pre-processing audio input to separate BGM with [[UVR>>url:https://github.com/Anjok07/ultimatevocalremovergui]].

35

* Post-processing with speaker diarization using the [[pyannote>>url:https://huggingface.co/pyannote/speaker-diarization-3.1]] model.

36

** To download the pyannote model, you need to have a Huggingface token and manually accept their terms in the pages below.

37

**1. [[https:~~/~~/huggingface.co/pyannote/speaker-diarization-3.1>>url:https://huggingface.co/pyannote/speaker-diarization-3.1]]

38

**1. [[https:~~/~~/huggingface.co/pyannote/segmentation-3.0>>url:https://huggingface.co/pyannote/segmentation-3.0]]

39

40

=== Pipeline Diagram ===

41

42

43

[[~[~[image:https://private-user-images.githubusercontent.com/97279763/389952408-1d8c63ac-72a4-4a0b-9db0-e03695dcf088.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDEzMDA1MTQsIm5iZiI6MTc0MTMwMDIxNCwicGF0aCI6Ii85NzI3OTc2My8zODk5NTI0MDgtMWQ4YzYzYWMtNzJhNC00YTBiLTlkYjAtZTAzNjk1ZGNmMDg4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAzMDYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMzA2VDIyMzAxNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWViOTYzZTdlYzdiYTRkMzYwY2YxZmZhOGM4Njc4OGNkMDFjYmIzZjMzNTdlYjA0ODUyZTZkNWZiNDQwZjlmMGQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.-7Zyn-HpREZdCWVjZ_hExH50vL-tSiCkxQmuBHgaJvE~|~|alt="Transcription Pipeline"~]~]>>url:https://private-user-images.githubusercontent.com/97279763/389952408-1d8c63ac-72a4-4a0b-9db0-e03695dcf088.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDEzMDA1MTQsIm5iZiI6MTc0MTMwMDIxNCwicGF0aCI6Ii85NzI3OTc2My8zODk5NTI0MDgtMWQ4YzYzYWMtNzJhNC00YTBiLTlkYjAtZTAzNjk1ZGNmMDg4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAzMDYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMzA2VDIyMzAxNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWViOTYzZTdlYzdiYTRkMzYwY2YxZmZhOGM4Njc4OGNkMDFjYmIzZjMzNTdlYjA0ODUyZTZkNWZiNDQwZjlmMGQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.-7Zyn-HpREZdCWVjZ_hExH50vL-tSiCkxQmuBHgaJvE]]

44

45

= Installation and Running =

* (((

== Running with Pinokio ==

50

)))

51

52

The app is able to run with [[Pinokio>>url:https://github.com/pinokiocomputer/pinokio]].

53

54

1. Install [[Pinokio Software>>url:https://program.pinokio.computer/#/?id=install]].

55

1. Open the software and search for Whisper-WebUI and install it.

56

1. Start the Whisper-WebUI and connect to the http:~/~/localhost:7860.

57

58

* (((

59

== Running with Docker ==

60

)))

61

62

1. Install and launch [[Docker-Desktop>>url:https://www.docker.com/products/docker-desktop/]].

63

1. Git clone the repository

64

65

{{{git clone https://github.com/jhj0517/Whisper-WebUI.git}}}

66

67

1. Build the image ( Image is about 7GB~~ )

68

69

{{{docker compose build }}}

1. Run the container

{{{docker compose up}}}

74

75

1. Connect to the WebUI with your browser at http:~/~/localhost:7860

76

77

If needed, update the [[docker-compose.yaml>>url:https://github.com/jhj0517/Whisper-WebUI/blob/master/docker-compose.yaml]] to match your environment.

* (((

== Run Locally ==

)))

=== Prerequisite ===

To run this WebUI, you need to have git, 3.10 <= python <= 3.12, FFmpeg.

87

And if you're not using an Nvida GPU, or using a different CUDA version than 12.4, edit the [[requirements.txt>>url:https://github.com/jhj0517/Whisper-WebUI/blob/master/requirements.txt]] to match your environment.

88

89

Please follow the links below to install the necessary software:

90

91

* git : [[https:~~/~~/git-scm.com/downloads>>url:https://git-scm.com/downloads]]

92

* python : [[https:~~/~~/www.python.org/downloads/>>url:https://www.python.org/downloads/]] **3.10 ~~ 3.12 is recommended.**

93

* FFmpeg : [[https:~~/~~/ffmpeg.org/download.html>>url:https://ffmpeg.org/download.html]]

94

* CUDA : [[https:~~/~~/developer.nvidia.com/cuda-downloads>>url:https://developer.nvidia.com/cuda-downloads]]

95

96

After installing FFmpeg, **make sure to add the FFmpeg/bin folder to your system PATH!**

97

98

=== Installation Using the Script Files ===

99

100

101

1. git clone this repository

102

103

{{{git clone https://github.com/jhj0517/Whisper-WebUI.git}}}

104

105

1. Run install.bat or install.sh to install dependencies. (It will create a venv directory and install dependencies there.)

106

1. Start WebUI with start-webui.bat or start-webui.sh (It will run python app.py after activating the venv)

107

108

And you can also run the project with command line arguments if you like to, see [[wiki>>url:https://github.com/jhj0517/Whisper-WebUI/wiki/Command-Line-Arguments]] for a guide to arguments.

= VRAM Usages =

This project is integrated with [[faster-whisper>>url:https://github.com/guillaumekln/faster-whisper]] by default for better VRAM usage and transcription speed.

114

115

According to faster-whisper, the efficiency of the optimized whisper model is as follows:

116

117

118

|openai/whisper|fp16|5|4m30s|11325MB|9439MB

119

|faster-whisper|fp16|5|54s|4755MB|3244MB

120

121

If you want to use an implementation other than faster-whisper, use ~-~-whisper_type arg and the repository name.

122

Read [[wiki>>url:https://github.com/jhj0517/Whisper-WebUI/wiki/Command-Line-Arguments]] for more info about CLI args.

123

124

If you want to use a fine-tuned model, manually place the models in models/Whisper/ corresponding to the implementation.

125

126

Alternatively, if you enter the huggingface repo id (e.g, [[deepdml/faster-whisper-large-v3-turbo-ct2>>url:https://huggingface.co/deepdml/faster-whisper-large-v3-turbo-ct2]]) in the "Model" dropdown, it will be automatically downloaded in the directory.

127

128

[[~[~[image:https://private-user-images.githubusercontent.com/97279763/389192679-76487a46-b0a5-4154-b735-ded73b2d83d4.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDEzMDA1MTQsIm5iZiI6MTc0MTMwMDIxNCwicGF0aCI6Ii85NzI3OTc2My8zODkxOTI2NzktNzY0ODdhNDYtYjBhNS00MTU0LWI3MzUtZGVkNzNiMmQ4M2Q0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAzMDYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMzA2VDIyMzAxNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTBlMmJkM2E2YWUyZWViMzliMmJlMTE4MjNjYzU3ZTUzOTI4Y2Y0N2YzNWNiYWE4ZjNkYWVlNzA5NjYzMzhiYzMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.uG4-xbp8-hrJGsuzEBqzvsdK5iAm1KcSatFSUhDJ2GI~|~|alt="image"~]~]>>url:https://private-user-images.githubusercontent.com/97279763/389192679-76487a46-b0a5-4154-b735-ded73b2d83d4.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDEzMDA1MTQsIm5iZiI6MTc0MTMwMDIxNCwicGF0aCI6Ii85NzI3OTc2My8zODkxOTI2NzktNzY0ODdhNDYtYjBhNS00MTU0LWI3MzUtZGVkNzNiMmQ4M2Q0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAzMDYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMzA2VDIyMzAxNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTBlMmJkM2E2YWUyZWViMzliMmJlMTE4MjNjYzU3ZTUzOTI4Y2Y0N2YzNWNiYWE4ZjNkYWVlNzA5NjYzMzhiYzMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.uG4-xbp8-hrJGsuzEBqzvsdK5iAm1KcSatFSUhDJ2GI]]

Wiki source code of Whisper-AI