==========
== wete ==
==========
============================================================================================

Advanced Audio Filters for VLC - GSoC21

This blog composes the work done on audio filters during GSoC 2021 by me, Metehan Arslan under mentorship of François Cartegnie and Jean-Baptiste Kempf.

Previous developer Vedanta Nayak had elementary informations on his blog, i recommend checking it out. He explained module structure, upmixing and work he done really well.

Get The Code From Here!

My Experience

Let’s take a quick look at my pre-GSoC background:

I was MS Windows user for 19 years; running Linux only on VM (excluding smartphones), only using for specific things like security tests. Interested in multimedia with few knowledge, fundamental knowledge about audio (avg. audiophile) and having DSP lessons from online courses. In programming experience: Trying to learn Go and Rust along with WASM, have a broken git knowledge, broken C knowledge (from school), 0.0 miliseconds experience with IRC or Open Source Organizations.

Before applying the GSoC i was trying to understand the codes because they look nothing like my wanky C projects. VideoLAN’s Hacker Guide1 along with VLC Documentation2 is a gem for getting in.

First i had to run upmix.c, it wasn’t working properly so my mentor François created pipeline for the filter (many thanks!).

Then i faced with continous distortion with .flac files, after rewriting and cleaning the code distortion disappeared with some algorithms, the reason behind that is subtraction operations with some channels. Apparently subtraction in channels causes problems. There’s a solution(?) for this: subtract the coefficients and multiple it with Lt+Rt but of course this is not the desired way since you don’t exactly get differences between channels.

My biggest challenge was (and still is) dolby.c code. This one also had pipeline problems and solution was different than upmix one. After pipeline is corrected i had much time playing with code and realized there are channel distribution for different layouts. For example if you want to experience DPL (L R C S) in your 5ch system, surround channel assigned to rear left and rear right channels. Or if you have 4ch layout without center channel (L R SL SR), Surround assigned to SL and SR without center.

DPL (Dolby Surround) and Dolby ProLogic II matrices were calculated from encoding matrix is wrong3, i corrected them and applied normalization.

Hilbert Transformation4 was applying to rear channels as a whole, it should be separately applied as +90 and -90 degrees for each of the dummy left and dummy right channels and these dummy channels should be operated for rearleft and rearright according to table. At first i tried to split the Hil functions as a +90 and -90, the result was same. Then i created xleft, xright, yleft and yright dummy channels to work on but these caused noise too, so i realized the problem was hilbert.c code. Rolled on Handbrake’s source code5 to find phase shifting algorithm, because handbrake can encode multi channel audio to 2ch Dolby Surround or Dolby Pro Logic II. Codes were too organized that i haven’t able to find it. Then headed to my current position, hilbert from FFmpeg6 7. I do have hopes for this time.

Why Hilbert? you may ask, because hilbert is capable of +90 and -90 (+270) degrees of phase shifting and mostly used in DSP applications.

Delay was also causing noise even it was turned off, setting its value to 24 was crashing vlc, so i tried to correct it, made it worse :D In the end stripped delay from the code and started implementing ring modulation delay effect. Suprisingly i haven’t got any errors on compile and i thought this shouldn’t be this easy: result was highly pitched sound with overdrive and cool ring modulator effect. This effect is great for driving instruments tho, code is here if you are interested. After trying to fix it i’ve lost and started working on chorus flanger delay.

I want to thank François Cartegnie, Jean-Baptiste Kempf, Alexandre Janniaux and Vedanta Nayak for help and support they provided.

What i’ve done?

  • Corrected the matrix parameters, applied normalization.
  • Added more upmix matrices.
  • Recycled/Rewrote the upmix.c; added prototypes, code is much more clean and readable now.

Thanks to Vedanta he already wrote the upmix.c easy to read but it’s form was not suitable for adding new matrices.

  • For preventing crashes i had to eject hilbert, it is now under maintenance.
  • Wrote and removed lines of failed codes along with tears.

Still in the works:

Phase Shifter

Phase shifter was applied wrong; current hilbert transform is causing distortion and crash, working on a new algorithm, in future this can be used in upmix.c for more upmix matrices. Also working on implementing DPLIIx and DPLIIz for 7ch and 9ch.

Delay

Rear channel delay had weird distortion, was causing crash and wasn’t efficient (lots of if else blocks) replaced it with ring modulator, performance was good but sound is janky, working on implementing chorus flanger delay.

Code Redundancy

I’ve done some cleanings but current code can always be written more clean and efficient.

PCA Upmix

Have problems running this, so i temporarily removed from the code, will continue after phase shift implementation.

To Do:

  • Low-pass filter for LFE channel.

  • Upmix matrix creation on gui.

  • Dynamic channel distribution for upmix.c (check conditions in dolby.c)

  • Fix flac noise issue.

How to add custom upmix matrix to upmix.c?

For now upmix.c channel count is statically 4, however if you want to increase, i assume you know already what you are doing :)

Now, head over to /modules/audio_filters/channel_mixer/upmix.c

First of all give your matrix a name, let’s name it after movie Matrix: Matrix Reloaded. Let’s add MATRIX_RELOADED codename inside enum,

enum
{
    PASSIVE_SURROUND,
    DOLBY_PROLOGIC,
    DOLBY_PROLOGICII,
    HAFLER_CIRCUIT,
    DYNAQUAD,
    ELECTROVOICE,
==> MATRIX_RELOADED,
};

Then add same codename to algo_list.

static const int algo_list[] = {
    PASSIVE_SURROUND    ,
    DOLBY_PROLOGIC      ,
    DOLBY_PROLOGICII    ,
    HAFLER_CIRCUIT      ,
    DYNAQUAD            ,
    ELECTROVOICE        ,
==> MATRIX_RELOADED     ,
};

Add string name to the algo_list_text[], this name will be visible on the GUI, put a comma and type your module string N_("Matrix Reloaded") and put it inside this list.

    static const char *const algo_list_text[] = {
    N_("Passive Surround Decoding"), N_("Dolby Pro Logic"), 
    N_("Dolby Pro Logic II"), N_("Hafler Circuit"), N_("Dynaquad"),
    N_("Electro-Voice"), N_("Matrix Reloaded")
};                     ~~~~~~~~~~~~~~~~~~~~~~~

Now if you compile, you can see your filter name under Preferences(All)>Filters>General Upmixing.

We are not done yet, head over to DoWork function and add your codename as a case like case MATRIX_RELOADED; under this you can add your matrix parameters according to template matrix i’ve provided down below.

    case MATRIX_RELOADED :
        msg_Dbg(p_filter, "Using Matrix Reloaded Upmix");
        f_0x0 =  0.9;     // Left Total Multiplier added to the Left channel, Default is 1.0
        f_0x1 =  0.1;     // Right Total Multiplier added to the Left channel, Default is 0.0
        f_1x0 =  0.1;     // Left Total Multiplier added to the Right channel, Default is 0.0
        f_1x1 =  0.9;     // Right Total Multiplier added to the Right channel, Default is 1.0
        f_2x0 =  0.66;     // Left Total Multiplier added to the 3rd channel
        f_2x1 = -0.33;     // Right Total Multiplier added to the 3rd channel
        f_3x0 =  0.33;     // Left Total Multiplier added to the 4th channel
        f_3x1 = -0.66;     // Right Total Multiplier added to the 4th channel
        break;
}
    !!! Do not exceed 1.0 value per channel total, this wil result with clipping. !!!

Template Matrix:

               S  O  U  R  C  E
    ___________________________
    |         |  Left | Right |
    |* * * * *|*******|*******|
S   | ch0 (L) |  0x0  |  0x1  |          A x B
    |- - - - -|-------|-------|         --   --
I   | ch1 (R) |  1x0  |  1x1  |          |   |
    |- - - - -|-------|-------|        sink  source
N   | ch2     |  2x0  |  2x1  |         nr     nr
    |- - - - -|-------|-------|
K   | ch3     |  3x0  |  3x1  |
    |_________|_______|_______|

If you are confused with AxB format, A stands for your channel number [0,3] and B stands for source audio channel (0 for left and 1 for right). After adding the parameters your filter is ready to listen.

IMPORTANT NOTE: channel order must be the following:

1-MIDDLELEFT, 2-MIDDLERIGHT, 3-REARLEFT, 4-REARRIGHT, 5-REARCENTER, 6-CENTER,
7-LFE (No use in here for now), 8-LEFT, 9-RIGHT

  1. Developers Guide, VideoLAN Wiki https://wiki.videolan.org/Hacker_Guide/ ↩︎

  2. Data Structures, VLC Doc https://videolan.videolan.me/vlc/annotated.html ↩︎

  3. Matrix Decoding, Wikipedia https://en.wikipedia.org/wiki/Matrix_decoder ↩︎

  4. Hilbert Transformation, Mathworks https://www.mathworks.com/help/symbolic/sym.htrans.html ↩︎

  5. Handbrake Source Code, GitHub https://github.com/HandBrake/HandBrake ↩︎

  6. Hilbert, FFmpeg https://www.ffmpeg.org/ffmpeg-filters.html#hilbert ↩︎

  7. FFmpeg Source Code, GitHub https://github.com/FFmpeg/FFmpeg or https://git.ffmpeg.org/gitweb/ffmpeg.git/tree ↩︎