Low Latency Causes Disconnect #1

New issue

Closed

opened 2022-04-04 13:45:37 +00:00 by raytiley · 2 comments

raytiley commented

2022-04-04 13:45:37 +00:00

(Migrated from github.com)

In browsing through the code it doesn't appear that the latency actually affects the configuration of the vosk server at all?

I've got debug logs if that might explain what's going on: https://gist.github.com/raytiley/c9f741093a78367a010606eba337a8ec

This is from the vosk server docker container.

INFO:root:Connection from ('172.17.0.1', 61632)

INFO:root:Config {'sample_rate': 48000, 'words': True}

LOG (VoskAPI:CompileLooped():nnet-compile-looped.cc:345) Spent 0.00932693 seconds in looped compilation.

ERROR:websockets.server:Error in connection handler

Traceback (most recent call last):

  File "/usr/lib/python3/dist-packages/websockets/protocol.py", line 827, in transfer_data

    message = await self.read_message()

  File "/usr/lib/python3/dist-packages/websockets/protocol.py", line 895, in read_message

    frame = await self.read_data_frame(max_size=self.max_size)

  File "/usr/lib/python3/dist-packages/websockets/protocol.py", line 971, in read_data_frame

    frame = await self.read_frame(max_size)

  File "/usr/lib/python3/dist-packages/websockets/protocol.py", line 1047, in read_frame

    frame = await Frame.read(

  File "/usr/lib/python3/dist-packages/websockets/framing.py", line 105, in read

    data = await reader(2)

  File "/usr/lib/python3.9/asyncio/streams.py", line 723, in readexactly

    await self._wait_for_data('readexactly')

  File "/usr/lib/python3.9/asyncio/streams.py", line 517, in _wait_for_data

    await self._waiter

asyncio.exceptions.CancelledError


The above exception was the direct cause of the following exception:


Traceback (most recent call last):

  File "/usr/lib/python3/dist-packages/websockets/server.py", line 191, in handler

    await self.ws_handler(self, path)

  File "/opt/vosk-server/websocket/./asr_server.py", line 38, in recognize

    message = await websocket.recv()

  File "/usr/lib/python3/dist-packages/websockets/protocol.py", line 509, in recv

    await self.ensure_open()

  File "/usr/lib/python3/dist-packages/websockets/protocol.py", line 812, in ensure_open

    raise self.connection_closed_exc()

websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

The command i'm using is:

gst-launch-1.0 vosk_transcriber latency=5000 name=tc ! fakesink sync=true dump=true uridecodebin uri=file:///e:/content/dn.mp4 name=decode ! queue ! audioconvert ! tc.

Hello! Giving this plugin a whirl and was curious about trying to acheive lower latency. When using default latency of 30s it seems to work fine. However if I go to anything lower the plugin runs for a bit, but then vosk eventually terminates the connection. In browsing through the code it doesn't appear that the latency actually affects the configuration of the vosk server at all? I've got debug logs if that might explain what's going on: https://gist.github.com/raytiley/c9f741093a78367a010606eba337a8ec This is from the vosk server docker container. ``` INFO:root:Connection from ('172.17.0.1', 61632) INFO:root:Config {'sample_rate': 48000, 'words': True} LOG (VoskAPI:CompileLooped():nnet-compile-looped.cc:345) Spent 0.00932693 seconds in looped compilation. ERROR:websockets.server:Error in connection handler Traceback (most recent call last): File "/usr/lib/python3/dist-packages/websockets/protocol.py", line 827, in transfer_data message = await self.read_message() File "/usr/lib/python3/dist-packages/websockets/protocol.py", line 895, in read_message frame = await self.read_data_frame(max_size=self.max_size) File "/usr/lib/python3/dist-packages/websockets/protocol.py", line 971, in read_data_frame frame = await self.read_frame(max_size) File "/usr/lib/python3/dist-packages/websockets/protocol.py", line 1047, in read_frame frame = await Frame.read( File "/usr/lib/python3/dist-packages/websockets/framing.py", line 105, in read data = await reader(2) File "/usr/lib/python3.9/asyncio/streams.py", line 723, in readexactly await self._wait_for_data('readexactly') File "/usr/lib/python3.9/asyncio/streams.py", line 517, in _wait_for_data await self._waiter asyncio.exceptions.CancelledError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/websockets/server.py", line 191, in handler await self.ws_handler(self, path) File "/opt/vosk-server/websocket/./asr_server.py", line 38, in recognize message = await websocket.recv() File "/usr/lib/python3/dist-packages/websockets/protocol.py", line 509, in recv await self.ensure_open() File "/usr/lib/python3/dist-packages/websockets/protocol.py", line 812, in ensure_open raise self.connection_closed_exc() websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason ``` The command i'm using is: `gst-launch-1.0 vosk_transcriber latency=5000 name=tc ! fakesink sync=true dump=true uridecodebin uri=file:///e:/content/dn.mp4 name=decode ! queue ! audioconvert ! tc.`

rafaelcaricio commented

2022-04-05 15:09:29 +00:00

(Migrated from github.com)

Hey, thank you for the interest in this work!

The latency configuration only affects when the sinks elements of the GStreamer pipeline are expected to start receiving buffers (and how much delayed). Honestly, the Vosk websockets server is not very reliable. In order to use this in production I would recommend doing a more reliable implementation on the server side of things. The protocol of communication (websockets/payload) could continue to be the same though, so no changes would be necessary in the vosk_transcriber element.

From the logs you shared, we can see the error:

0:00:50.269486000 20436 00000256C72725C0 ERROR       vosk_transcriber src\transcriber\imp.rs:674:gstvosk::transcriber::imp:<tc> Failed sending audio packet to server: IO error: An established connection was aborted by the software in your host machine. (os error 10053)

Which points to some problem with the Vosk Server. Then on the server side what is interesting is this: connection closed abnormally [internal]. I guess would be nice to investigate what this "internal" means.. maybe there is a way to expose what is the actual error here.

Hey, thank you for the interest in this work! The latency configuration only affects when the sinks elements of the GStreamer pipeline are expected to start receiving buffers (and how much delayed). Honestly, the Vosk websockets server is not very reliable. In order to use this in production I would recommend doing a more reliable implementation on the [server side](https://github.com/alphacep/vosk-server/blob/3ac5120aa6b6f87be2a0fc8525a0ee6da0a12fc9/websocket/asr_server.py#L36) of things. The protocol of communication (websockets/payload) could continue to be the same though, so no changes would be necessary in the `vosk_transcriber` element. From the logs you shared, we can see the error: ``` 0:00:50.269486000 20436 00000256C72725C0 ERROR vosk_transcriber src\transcriber\imp.rs:674:gstvosk::transcriber::imp:<tc> Failed sending audio packet to server: IO error: An established connection was aborted by the software in your host machine. (os error 10053) ``` Which points to some problem with the Vosk Server. Then on the server side what is interesting is this: `connection closed abnormally [internal]`. I guess would be nice to investigate what this "internal" means.. maybe there is a way to expose what is the actual error here.

rafaelcaricio commented

2022-04-12 10:44:58 +00:00

(Migrated from github.com)

Hey @raytiley ,
I have looked more deep into this issue and it seems to only happen when running the pipeline using the gst-launch-1.0 command. If you write a small application and use the plugin this problem does not happen. Of course the transcription will not come in time and the pipeline will give warnings that many buffers are being dropped, etc.. but it does not drop the connection.

If you are looking for a working example to play around, try this https://github.com/rafaelcaricio/example-add-captions708/blob/vosk-local-transcription-demo/src/main.rs

I hope that helps.

Hey @raytiley , I have looked more deep into this issue and it seems to only happen when running the pipeline using the `gst-launch-1.0` command. If you write a small application and use the plugin this problem does not happen. Of course the transcription will not come in time and the pipeline will give warnings that many buffers are being dropped, etc.. but it does not drop the connection. If you are looking for a working example to play around, try this https://github.com/rafaelcaricio/example-add-captions708/blob/vosk-local-transcription-demo/src/main.rs I hope that helps.