LowRankLayers

LoRADense(primary::Dense, hidden_dim::Int; init=Flux.kaiming_uniform())

Create a LoRA wrapper around a Dense layer. The second projection matrix is initialized to zero, and only the two projections (and not the primary layer) are trainable.