Why does the Ubuntu Forums use proprietary software(most particularly vBulletin)?

vBulletin is proprietary, that is an accepted fact, but this doesn't really mean it is completely closed source. This is because if you purchase a license you get 100% code and you can make modifications to any of it as long as its not re-distributed.

If you require more information about vBulletin then you can refer to it's FAQ which can be found here:-
vBulletin FAQ

A few reasons why vBulletin was chosen over other free and open source software:-

1. vBulletin has a good reputation for being secure.

2. vBulletin is provided with all source so modifications can be made as needed.

3. vBulletin has functions other forum software do not have and those features will be implemented on this forum in the near future.

4. vBulletin has been integrated with the Ubuntu mailing lists.

5. This forum has a large database, a very active community and also requires certain features which cannot be fully catered to by other free and open source forum software. vBulletin on the other hand has those required features and the stability to handle the daily loads and the large database.

Did we ever try using free and open source alternatives?

Yes we did. The forums did run phpBB when it was first started yet phpBB was dropped as it did not meet the expectations that were required to run the Ubuntu Forums consistently.