Artificial Kuramoto Oscillatory Neurons

Title: Artificial Kuramoto Oscillatory Neurons
Authors: Takeru Miyato, Sindy Löwe, Andreas Geiger, Max Welling
Published: 17th October 2024 (Thursday) @ 17:47:54
Link: http://arxiv.org/abs/2410.13821v1

Abstract

It has long been known in both neuroscience and AI that “binding” between neurons leads to a form of competitive learning where representations are compressed in order to represent more abstract concepts in deeper layers of the network. More recently, it was also hypothesized that dynamic (spatiotemporal) representations play an important role in both neuroscience and AI. Building on these ideas, we introduce Artificial Kuramoto Oscillatory Neurons (AKOrN) as a dynamical alternative to threshold units, which can be combined with arbitrary connectivity designs such as fully connected, convolutional, or attentive mechanisms. Our generalized Kuramoto updates bind neurons together through their synchronization dynamics. We show that this idea provides performance improvements across a wide spectrum of tasks such as unsupervised object discovery, adversarial robustness, calibrated uncertainty quantification, and reasoning. We believe that these empirical results show the importance of rethinking our assumptions at the most basic neuronal level of neural representation, and in particular show the importance of dynamical representations.

---

Aim: Construct a different neuron component to allow to closer mimicking of McCullogh-Pitts neurons — fire with spatiotemporal dynamics $\to$ neurons that fire together wire together

• New kind of neuron to place in NNs - based on Artificial Kuramoto Oscillators
  • “Oscillators” are n-dimensional unit vectors $⟺$ vectors on the unit (hyper)sphere

Differential eqn. of the Kuramoto model:

$${\dot{\theta }}_i={\omega }_i+\sum _{j=1}^N{J}_{ij}\sin ({\theta }_j-{\theta }_i)$$

where:

• ${\theta }_i$: the phase of the $i$-th oscillator
• ${\omega }_i$: natural frequency of the $i$-th oscillator
• $N$: number of oscillators
• $J_{ij}$: coupling strength between the $i$-th and $j$-th oscillators 👈 this is the important one and $K>0⟹\text{synergistic}$; $K<0⟹\text{antagonistic}$

The Kuramoto model from Chemical Oscillations, Waves, and Turbulence SpringerLink is 1-dimensional and the representation of an oscillator is its phase information

In the above $\sin ({\theta }_j-{\theta }_i)$ represents the phase difference between the $j$-th and $i$-th oscillators

The differential eqn. for their vector-valued Kuramoto model:

$$\dot{\mathbf{x}}={\omega }_{\mathbf{i}}\cdot {\mathbf{x}}_{\mathbf{i}}+{\text{Proj}}_{{\mathbf{x}}_{\mathbf{i}}}\left({\mathbf{c}}_{\mathbf{i}}+\sum _j^N{\mathbf{J}}_{\mathbf{ij}}\cdot {\mathbf{x}}_{\mathbf{j}}\right)$$

where:

• $\mathbf{omeg}{\mathbf{a}}_{\mathbf{i}}\in {\mathbb{R}}^{N\times N}$ is an $N\times N$ anti-symmetric matrix an anti-symmetric matrix is one where $A=-A^T$ so basically like a symmetric one with the lower triangle multiplied by $-1$
• the $\dot{\mathbf{x}}={\omega }_{\mathbf{i}}\cdot {\mathbf{x}}_{\mathbf{i}}$ component is the oscillators natural frequency
• defn. ${\text{Proj}}_{{\mathbf{x}}_{\mathbf{i}}}(.)$ projects input vector ${\mathbf{x}}_{\mathbf{i}}$ onto the hypersphere’s tangent space - see Vector Projection