neural networks & ai

summer camp @ constructor school, together with Meri Grigoryan

neural ordinary differential equations

by Nuray Huseynli

The goal is to explain how Neural ODEs unify calculus and AI by replacing traditional neural network layers with continuous dynamics modeled by ordinary differential equations.

deliverables

1. From Layers to Continuous Dynamics

  • Explain why traditional deep learning uses discrete layers and how ODEs offer a continuous alternative.
  • Contrast ResNet‑style “skip connections” with Neural ODEs’ time‑continuous transformations.

Requirements

  • Define what an ODE using calculus terms.
  • Explain what layers are in traditional neural networks.
  • Describe why neural networks use discrete layers.
  • Explain ResNet skip connections and the advantage of Neural ODEs’ continuous transformations.

2. The Calculus Behind Neural ODEs

Understand how continuous models are trained via gradient‑based optimization.

Requirements

  • Use a standard ODE solver (e.g. torchdiffeq’s odeint) to simulate a simple neural ODE model.
  • Explain the concept of backpropagation through an ODE using the adjoint sensitivity method.
  • Provide an intuitive derivation of the adjoint method, emphasizing connections to the chain rule and reverse‑mode differentiation.
  • Bonus. Visualize how gradients evolve in a Neural ODE vs. a standard residual network.

3. Reflection

Analyze real‑world use cases, benefits as well as limitations.

Requirements

  • Improvements over the classical neural network scheme: Backpropagating through an ODE solution involves solving a second ODE backward in time, rather than storing intermediate activations in memory.
  • Limitations on computational cost, tradeoffs vs. CNNs/RNNs1.
  • Bonus. Time‑series forecasting (e.g., stock prices, COVID spread), generative modeling (e.g., continuous Normalizing Flows).
Resources

  1. Convolutional Neural Networks / Residual Neural Networks. ↩