The problem with restricting calories too much is that you start to starve the muscle and that has the opposite effect of what you want to accomplish. 2300 calories is not a lot of food, and if you're doing the workout and cardio you need everybit of that to maintain and build that muscle. Just make sure you're eating the correct foods, from everything I've read, that is honestly 80% of the battle.