I started using Team Foundation Service from its preview days when it used to be called TFS Preview. When I first got the invite for the service first thing I did was to create a Team Project and try and explore various sections within the service. Back then I just assumed that I will have a delete project option somewhere in the site, turns out you don't get the delete button :)
One of the thing that kept bothering me was some of the Team Project names that I added back then and I really wanted to get rid of them.
After looking around it turns out you can delete a team project using your normal delete team project command from developer command prompt.
TfsDeleteProject /collection:https://mytfs.visualstudio.com/DefaultCollection MyProject
And bang! your project is gone!