Why use SiLU before residual connection

[j93hahn]

Title. In

pykan/kan/KANLayer.py

Line 65 in ef4861e

def __init__(self, in_dim=3, out_dim=2, num=5, k=3, noise_scale=0.1, scale_base=1.0, scale_sp=1.0, base_fun=torch.nn.SiLU(), grid_eps=0.02, grid_range=[-1, 1], sp_trainable=True, sb_trainable=True, device='cpu'):

, why do you apply SiLU on the input x before passing it through the linear layer and then the residual connection? Why not the linear layer first then SiLU? Or why even use SiLU altogether?

[LarsDoorenbos]

Related: #128