neural network based pre-fft

🙋 If you want a new feature or a change to the software then add your request here.

If it's not here then it's not going to happen.
rainstar
Posts: 66
Joined: Sat Dec 19, 2020 4:06 am

neural network based pre-fft

#1

Unread post by rainstar »

So i admit i dont really know how sdrs or sdr software works but there is data in a “form” that sdrconsole then decodes and renders into visible and audible forms.
That original stream of bits proper prior to any FFT processing is what neural networks work best with and will do the best job of removing the noise by operating on that data- ie i think it is called IQ data or a “spectrum recording”?

Regardless to make it work we would need three things:
an API with input and output so we could train and make a modular system
a tool to combine multiple overlapping recordings so that we can overlay noise signals onto sample good signals
a tool to generate fake noise that acts just like real noise(we could also record some).

Its more a project than a request but it would make me very happy as around sundown on lower Hf above 80 meters everything goes to heck for several hours due to lightening storms hundreds of miles away but its all transient noise and that makes me think it could be removed, and even better so- a neural network understands what good signals behave like and what noise behaves like and so could possibly result in improved overall SNR.
just a thought

User avatar
PD3LK
Posts: 517
Joined: Thu Aug 06, 2020 10:36 am

Re: neural network based pre-fft

#2

Unread post by PD3LK »

Hi, the settings can be critical (depending on the QRM) but i find the Wide Band DSP Noise Blanker very usefull.

rainstar
Posts: 66
Joined: Sat Dec 19, 2020 4:06 am

Re: neural network based pre-fft

#3

Unread post by rainstar »

The noise blanker cannot reconstruct signal. A neural network can. A neural network could be trained on both noise-free as well as noise-polluted signals and, using a mode selector,
actively anticipate and cancel out the noise.

rainstar
Posts: 66
Joined: Sat Dec 19, 2020 4:06 am

Re: neural network based pre-fft

#4

Unread post by rainstar »

Neural-Network based IQ data noise removal, signal reconstruction, upsampling/anti-aliasing, ghost extraction, and Phase Error correction

Goal 1: to be able to apply synthesized gain to faint signals embedded in the noise floor for which non-obvious patterns exist in the sample data,
potentially allowing access to signals that would otherwise be unreadable

Goal 2: to be able to take samples containing carriers and signal and adjust them to reduce the phase error

Goal 3: to be able to take an input signal and process it using a lookahead buffer to remove multiple kinds of noise pending option selection:
wideband pulse, sweeping pulse, random noise, weather static, and to be able to reconstruct or hallucinate signal details to fill in the blank up to
audible accuracy for any period under 160ms

Goal 4: to be able to upsample 16bit samples to 32 with reasonable accuracy using zoom/enhance hallucination techniques

Goal 5: to combine all four stages of neural network pre-filtering/compensation into one module that uses openCuda and dedicated hardware to real-time
process input signals with 130ms buffer before sending it on


Neural-Network based pseudo-FFT voice noise removal, signal reconstruction, upsampling/anti-aliasing, ghost extraction, sampling method
Take all of the lessons learned in the data/signal correcting neural module and apply it to a single voice signal in SSB

Note this is NOT synonymous with the openrnn based noise reduction systems. Those are designed to hallucinate voice patterns and extract them, and are based on existing methods
from the mid 2000s and early 2010s. Advances in neural network design have demonstrated that exponentially better increases in performance are possible.
https://authors.library.caltech.edu/106 ... -Paper.pdf
In particular the ability of neural networks to utilize buffers of previous data to enhance lookahead prediction as well as buffers of upcoming data would be used for this project,
in both the signal processing and the voice processing stages. Essentially a matrix of inputs would be used to help the network make good judgements.

Goal 1: to be able to apply synthesized gain to faint signals embedded in the noise floor for which non-obvious voice modulation patterns exist in the sample data,
potentially allowing access to signals that would otherwise be unreadable

Goal 2: to be able to take samples containing multiple overlapping echo/phase shifted reflections and shift each so they constructively interfere and resolve
into one stronger signal

Goal 3: remove noise from voice signals using anticipated waveforms and trained neural network models

Goal 4: to apply additional gain, clarity, reverb, dynamic adjustment, and EQ to voice signals automatically

Goal 5: to rectify the voice into an output buffer using a synthetic FFT-emulating network capable of automatically expanding/contracting to the waveform

Goal 6: to combine all five stages into one module that acts like the data module and takes an input buffer and outputs it into the SDR software pipeline as audio

-----------

Goal 7: to combine the data and voice modules into one system

User avatar
Simon G4ELI
Posts: 2134
Joined: Thu Aug 06, 2020 7:27 am
Location: Mawnan Smith
Contact:

Re: neural network based pre-fft

#5

Unread post by Simon G4ELI »

Just FWIW NR3 is neural network based.

https://jmvalin.ca/demo/rnnoise/
Simon Brown :shock:
www.sdr-radio.com

Do not send me direct e-mail, thank-you!

rainstar
Posts: 66
Joined: Sat Dec 19, 2020 4:06 am

Re: neural network based pre-fft

#6

Unread post by rainstar »

Even so the RNNoise option you ship is not optimal: it is inferior to the one shipped with OBS, for reasons I don't understand. Might be a newer model of the net training data?
Neural networks also seem to do better with analogue inputs. Feeding, for example, gnif audio filter(an optimized version of rrnoise trained on 10gb of data) with inputs using various windowing functions
resulted in different outputs. This is most visible if you *obtain* *cough cough* a copy of- and i dare not mention which russian helped me- Accentize VoiceGate.
Accentize VoiceGate, a very expensive but not better tool, has a realtime visualization component that shows where the neural network was able to "see" voice activity and where it does not.
AM radio, for example, because of the music, makes it hard for the neural network to extract the spectral components of voice activity- because it's not trained to recognize music as noise.
Depending on the windowing function used and at what stage I applied what noise reduction techniques like equalizers, FFT's, gain- I was able to enhance the ability of the network to recognize voice.

Even so, it has to take the input from a shaped output. It doesn't see the IQ data stream, only what the windowing function has rectified. And ironically, FFT's behave like neural networks.
https://sidsite.com/posts/fourier-nets/
https://dawn.cs.stanford.edu/2019/06/13/butterfly/
all matrices admit a representation as a composition of only sparse butterfly matrices and permutation, is a conclusion the stanford researchers came to.

Any neural network can be trained to perform the transformations that a windowing function does, along with EQ and high/low pass at the same time, and to work as well as a hand optimized function.
Your brain does all kinds of things continually better than any computer algorithm, and it's mostly a self-trained network.

Ideally, we could replace *all* parts of the software pipeline after the ADC output with neural network components, because they can be better optimized to understand how the data presented interacts.

User avatar
Simon G4ELI
Posts: 2134
Joined: Thu Aug 06, 2020 7:27 am
Location: Mawnan Smith
Contact:

Re: neural network based pre-fft

#7

Unread post by Simon G4ELI »

What is OBS?

It's difficult to train rrnoise as it should be left adaptive I think but one thing I want to investigate is whether I can train rrnoise with old data.
Simon Brown :shock:
www.sdr-radio.com

Do not send me direct e-mail, thank-you!

rainstar
Posts: 66
Joined: Sat Dec 19, 2020 4:06 am

Re: neural network based pre-fft

#8

Unread post by rainstar »

OBS is the open broadcaster suite, a modular program that people use to stream to youtube, to record outputs, and as a virtual camera application. it is extremely powerful and versatile.
I am able to pipe sdr-console to VB-Cable -> VSTHost.Exe(still one of the best programs around, can load up vst 2/3 and use them to process audio no problem) -> OBS and stream what I hear.
I am also able, if the audio buffer latency is consistent to delay the video in OBS- I'm able to sync them perfectly going out to youtube. Streaming my radio with less than 3 second latency.
RNNoise training, like all neural networks, requires a combination of both clean data and noisy data, and getting good noisy data is somewhat harder than I imagined.
The neural network is optimized to recognize what "this is good signal" looks like, and can learn what shortwave, modems, codar, SSB, dual sideband, time signals,
AM, VLF- it can be trained on all kinds of signals and this is good and fine.

However, putting noise into the signal which isn't just static or non-rf noise is hard. Simulating BAD signal on top of GOOD signal is challenging.
just adding white or brown noise or outdoor recordings doesnt do anything for the neural network to teach it how to work.

I spent a lot of time trying to phase out noise for my own experimenting and think there's a method to this.

Ideally, we'd have a three antenna setup- one is 20db lower noise(I know a man working on a technique he claims can do this through infinite space point compression, how I dont know)
one antenna which is somewhat noisy and picks up noise and signal equally, and finally a random wire antenna which is grounded and randomly polarized which picks up nothing but nearby noise.
The first antenna's output gets subtracted from the second to produce source C which could be called "a realistic diversity of noise sources present in the environment"
then the random wire samples are gate-ducked to combine them at same gain levels, which produces a source D "a diversity of noise sources INCLUDING noise sources which underly signal-
aka reconstruct the noisy antenna and add back to it whatever the good antenna takes out, which is actually noise quanta that the good antenna picks up.

Then we have a very noisy sample with nothing useful in it, a very good sample with only good signals in it, and a sample which is realistic.

So far, we train the network on the last looking for the second- but neural networks which are over-trained often have loss characteristics associated with training for local quanta.
In short, they become less effective as they learn really well how to remove what the discriminator is looking for, and plateau.

In the mode I suggest, the mixture between the first and last(true noise + noisy) is adjusted from a ratio of 0:1 to 1:0 over the course of training.
So, finally, we have three samples: a sample which is clean, a sample which is dirty, and a sample which is junk.
We create samples for all three across a full 24 hour period during good band conditions, in a controlled environment, using a synced array of receivers that are 16 bit.
Each sample contains the exact same audio at the exact same time.

The neural network is then fed the dirty and junk samples at random in a 50:50 ratio, with the discriminator looking for the clean signals.
Initially, it will become more able to solve the dirty -> clean but as it over-trains it will try to optimize but the junk samples will prevent this-
unless it figures out some crazy way to get signal out of the noise floor, which would also be great.

Post Reply