Research InterestsWe solve problems in machine learning caused by finite-precision arithmetic, the arithmetic computers actually use. Our main tool is machine learning theory, which asks why modern machine learning works, using mathematics: what functions can a model compute (expressivity)? Why does training find good parameters (optimization)? And why does a trained model work on data it has never seen (generalization)? Why Finite-Precision Arithmetic?Computers do not compute with real numbers. Mathematics works with infinitely many real numbers and exact operations. A computer does not: it uses finite-precision arithmetic, such as floating-point arithmetic, which can represent only finitely many numbers, and almost every operation rounds its result to a representable number. Machine learning theory, however, is usually developed under ideal real-number arithmetic, while every deployed model runs on finite-precision arithmetic. A central question of our lab is: Do the conclusions derived under idealized mathematics survive on a real computer? The answer is often no, and in surprising ways: rounding is not merely a small error added at the end, but a change in the model of computation itself. Finite precision changes what networks can express. Consider a neural network whose activation function is the identity function, so that the network does nothing but compose affine maps. Under real arithmetic, such a network can compute only affine functions. Under floating-point arithmetic, the same architecture can approximate essentially arbitrary functions, because the rounding in each operation itself acts as a nonlinearity. Results of this kind show that the expressive power of a network on a real computer can differ fundamentally from its expressive power under real arithmetic. We have characterized which functions floating-point networks and transformers can represent, and how much depth and width they need to do so. Training and generalization change as well. Learning algorithms are also executed in finite-precision arithmetic. The gradients used in training are produced by automatic differentiation, which does not always compute the true mathematical gradient; we study what it actually computes, and when and why it is correct. Moreover, convergence guarantees for optimization algorithms and generalization bounds for the trained models, proven under exact arithmetic, can break or degrade once rounding is taken into account. We develop theory for training and generalization that holds under finite precision. The gap creates security, privacy, and verification problems. A privacy mechanism that is provably private in the idealized model can leak private data through its rounding patterns. A model certified to be robust against adversarial inputs can still be attacked when it runs in floating point. And a formal proof about a mathematical model of a network may not hold for the network that actually runs. We study when such failures happen and how to prevent them, aiming at guarantees for the computation that is actually performed rather than for its idealized model. Current Directions
See Publications for our results on these questions, such as how narrow a network can be while remaining a universal approximator, what automatic differentiation actually computes for neural networks, and what floating-point networks can and cannot represent. If these questions interest you, see How to Join. |