Brillouin light scattering is an established technique to study magnons, the elementary excitations of a magnet. Its efficiency can potentially be enhanced by cavity effects that concentrate the light power in the magnet. Here, we study inelastic scattering of photons by a magnetic sphere that supports optical whispering gallery modes, in a configuration of light traveling perpendicular to the magnetization. We find light scattering in two regimes. For low angular momentum magnons, the light is scattered in the forward direction with a pronounced asymmetry in the Stokes and the anti-Stokes scattering probability, consistent with recent experiments. High angular momentum magnons back-scatter light into either the Stokes or anti-Stokes peaks. We further show that the light scattering in the latter regime permits mapping of the high angular momentum magnon dispersion.