I'm trying to do this as well. Furthest I came was this ominous line in the xfce documentation:
You can set a custom icon size in gtk-icon-sizes with the name panel-tasklist-menu. The default icon size is 16px.
I tried to do this by changing
<property name="IconSizes" type="empty" />
<property name="IconSizes" type="string" value="gtk-menu=32,32:gtk-button=32,32:panel-applications-menu=32,32:panel-directory-menu=32,32:panel-tasklist-menu=32,32"/>
Effectively changing the size of most of the icons except those in my panel...