I had blacklisted the B43 but it keeps showing up in the Hardware Drivers and what appears that it continues to be used despite my attempts to uninstall in Synaptic and only install the BCM43xx in Synaptic. Funny thing is that it may be using the BCM43xx that is installed as well because the issues has much improved (see Below)

5776 packets transmitted, 5599 received, 3% packet loss, time 28274710ms
rtt min/avg/max/mdev = 0.277/2.677/884.216/33.777 ms