As theoretical studies suggest the generalization power of deep neural networks comes from the expressivity of their non-linearities. However, it has been shown recently that in practice deep linear networks are also able to achieve performance on par with their non-linear analogs. In his research, Igor investigates how incorporating non-linear interactions into the deep neural networks affects their expressivity both in theory and in practice. In particular, he explores the role of multiplicative interactions in shaping the generalization of non-linear layers, utilizing challenging long-term memorization tasks for evaluation.