VLC.js Audio Pipeline - GSoC22

Hello!

In this page you will find the work done in VLC.js Audio Pipeline during Google Summer of Code 2022 by Metehan Arslan; under mentorship of Alaric Sénat, Jean-Baptiste Kempf and Mehdi Sabwat.

There is two repositories for the audio output, one of them was written in unified cpp file written in C++&JS hybrid. Other repository was C with JS. I am more comfortable with C than C++, so i worked on this repository.

Get the Code!

What I've done:

• Fixed support for formats that frame size isn't exact divisor of 1024*1024.

• Downmixing in VLC-side.

• Misc. deprecation updates.

• Tiny UI changes.

To Do:

• Fixing glitches/bugs in progress bar.

• Audio Filter Support.

• Better UI&UX with detailed menus, filters and configurations.

Important Note: Currently (September 2022) Gecko based browsers does not have stable multichannel audio playback. I recommend to use Chromium based browsers for better overall experience.

Fixed Support For Formats That Frame Size Isn't Exact Divisor of 1024*1024

I've noticed this bug while playing the flac files. Actually we were able to play flac files but there was clicking/popping noise in every 3 seconds.

#define STORAGE_SIZE=1024*1024, Mehdi well explained why it's 1024*1024:¹

The audio callback will consume 128 samples each time process() is called, so having less than 128 samples might block audio_worklet_push() for too long. it was 1024 initially and then I think the idea was to have a bigger size to allow rendering enough samples.

I thinked about defining STORAGE_SIZE with specific vaule that divisor of many samples, but i wasn't sure how efficient this was; Since there is various audio formats and we are working on VLC, which should play everything :)

Then my mentors pointed that ring buffer dividing the data if incoming data is bigger than storage size.

The problem contained in ring buffer. Can you address the problem?

    if (head + data_size > STORAGE_SIZE)
    {
        // Copies the part of the data at the buffer end
        unsigned data_size_copy_end = STORAGE_SIZE - head;
        memcpy(sab_view + head, buffer, data_size_copy_end);
        head = 0;

        // Copies the part of the data at the buffer start
        unsigned data_size_copy_start = data_size - data_size_copy_end;
        memcpy(sab_view + head, buffer, data_size_copy_start);
        head = data_size_copy_start;
    }

Fix

2nd memcpy() were copying the same buffer. I took me a while to see this typo

    if (head + data_size > STORAGE_SIZE)
    {
        // Copies the part of the data at the buffer end
        unsigned data_size_copy_end = STORAGE_SIZE - head;
        memcpy(sab_view + head, buffer, data_size_copy_end);
        head = 0;

        // Copies the part of the data at the buffer start
        unsigned data_size_copy_start = data_size - data_size_copy_end;
        memcpy(sab_view + head, buffer + data_size_copy_end, data_size_copy_start);
        head = data_size_copy_start;   `~~~~~~~~~~~~~~~~~~~`
    }

Downmixing in VLC-side

We were having problems with non-native playback. I thought always passing file's channel count directly and leave every other thing to browser² was a good idea--I was wrong.³

So i started working on to downmix the channels. In order to prevent spaghetti if-else conditions, created an array:

int channel_layout[] = {AOUT_CHANS_2_0, AOUT_CHANS_2_0, AOUT_CHANS_2_0,
        AOUT_CHANS_3_0, AOUT_CHANS_4_0, AOUT_CHANS_5_0, AOUT_CHANS_5_1, 
        AOUT_CHANS_7_0, AOUT_CHANS_7_1, AOUT_CHANS_8_1};

...

sys->channels = aout_FormatNbChannels(fmt);
sys->webaudio_channels = webaudio_getChannels();

...

if (sys->webaudio_channels < sys->channels)
{
    sys->channels = sys->webaudio_channels;
    msg_Err(aout, "sys after downmix %u", sys->channels);
}

fmt->i_physical_channels = channel_layout[sys->channels];
fmt->i_channels = sys->channels;

As you can see we are getting the file's channel count with aout_FormatNbChannels(fmt), comparing the file and output channel counts then downmix. But currently we cannot reach webaudio_getChannels() in Start() function.

webaudio_getChannels: function() {
    if (Module.webaudioContext === undefined)
        return 0;
    return Module.webaudioContext.destination.maxChannelCount;
}

Another problem is replaying after playback results with playing again in slow-motion. The reason is probably related to channel count, ring buffer reads more that 2 channel data as 2 channel.

Misc. Deprecation Updates

Updated some expressions in vlc.js emsdk tools and C/C++ code.

Tiny UI Changes

Time text is more beautiful now, clicking to progress bar now plays the content if paused. Progress bar had bugs so i tried to add overlapping bar but it just was appearing random places so i had to remove it :D Added metadata prototypes but it's not ready yet.

Fixing Glitches/Bugs in Progress Bar

As mentioned above, progress bar behaves unexpected while paused, time text stays behind (And so is progress bar). Volume bar is also acting quite late. I think some of the UI should rewritten as atomic, some of the values like get_time or get_position is handled atomically and have specific conditions while paused.

Audio Filter Support

Currently wasn't able to run filters, when arguments typed in the box and send; got errors saying i typed weird characters. Maybe some overflow/underflow going on?

Better UI&UX With Detailed Menus, Filters and Configurations

It would be great if we were get close to VLC's desktop experience with extended options.