LowRankLayers
Documentation for LowRankLayers.
LowRankLayers.LoRADense
— MethodLoRADense(primary::Dense, hidden_dim::Int; init=Flux.kaiming_uniform())
Create a LoRA wrapper around a Dense layer. The second projection matrix is initialized to zero, and only the two projections (and not the primary layer) are trainable.