After extensive usage, I decided to purge linux-firmware-nonfree and install the Broadcom STA drivers through jockey.

I'm not sure why the STA drivers did not work intially but everything seems to be fine now.

Also my interfaces looks like this :

auto lo
iface lo inet loopback
And I deleted everything in b43-pci-bridge.conf