On the number of linear regions in a Rectified Network

Details

Supervisors

Hendrickx, Julien

Faculty

Ecole polytechnique de Louvain

Degree label

Master [120] en sciences informatiques, à finalité spécialisée

Abstract

enNeural networks have become ubiquitous in our modern world. This widespread success can largely be attributed to advances in deep learning, a field that has demonstrated a remarkable capacity for universal learning across a wide range of tasks and domains. The way deep neural networks do this is not well understood, but for a certain class of neural networks, known as rectified networks, which possess piecewise linear activation functions, the expressivity can be linked to the number of linear regions they partition the input space into. Several works have derived upper bounds on the number of regions a rectified network can achieve and a formula for the expected value was conjectured in 2019. In this work we prove this formula, showing it holds under mild assumptions. This formula does not depend on the number of layers in the network, only the number of neurons, which is in conflict with the long-standing assumption that deep networks are exponentially more expressive than shallower ones. We demonstrate both theoretically and experimentally that this is not case.