Um, if you write drivers you are a kernel developer by definition, since drivers go in the kernel.
There are hundreds (thousands?) of people contributing to the kernel, not to mention many major corporations. There are probably more kernel developers than Canonical employees. So many that Linus has started telling people to go work on other open source projects, because the kernel has more than enough developers. The "Linux Drivers Project" (a special project dedicated solely to advancing Linux hardware support) itself boasts four hundred developers last time I checked.Such a large task for such few people.
See http://www.linuxdriverproject.org for more information on it.