Finally got it working. I have a Netgear router with DD-WRT firmware. Here is what fixed it for me:
I had to forward each port manually. Even when manually forwarding ports I could not use the 'both' setting. I had to forward udp and tcp manually for each port. so I have a total of 4 forwarding rules set up for my xbox in the dd-wrt settings. 2 for udp and 2 for tcp. worked fine then. weird. Must be something wrong with the 'both' setting for my dd-wrt firmware???