We give a geometric construction of the phase space of the elliptic Calogero-Moser system for arbitrary root systems, as a space of Weyl invariant pairs (bundles, Higgs fields) on the $r$-th power of the elliptic curve, where $r$ is the rank of the root system. The Poisson structure and the Hamiltonians of the integrable system are given natural constructions. We also exhibit a curious duality between the spectral varieties for the system associated to a root system, and the Lagrangian varieties for...