Oblivion uses the Xbox 360 hard drive to cache game data and in some cases the info stored on the hard drive can get fragmented, slowing down the game's performance. Bethesda has already found out a way to fix the issue. Here is the official word:

Oblivion uses the Xbox 360 hard drive extensively to cache (copy and reuse) game data. This is done to optimize all loading the game does. It has been found in rare cases on some Xbox 360s that the cached data can get overly fragmented. This exhibits itself by the game taking a long time to load anything from menus, dialogue, sounds, objects, levels, and more.

To solve this problem, reset your Xbox, and hold down the A button as the game is booting up. If any button is held while Oblivion is loading, it will clear its hard drive cache, and create a new one.

Once you see the “Bethesda Softworks” logo video, the game is already loaded, and the cache should have been cleared.

Note, this does not effect your game in any way other than loading. You will not lose any saved data.