no, console-conf is the tool that comes up on screen to set up network and the user, you should not disable it for now if you want it to be able to ssh into the device later and it is also a good step to see everything is working as expected … just focus on getting your driver to work in the kernel and make sure your gadget is correct so snapd can do what it needs to set the right cmdline options.
Additionally, I tried kernel and dtb from kernel snap with yocto filesystem and tested ping. That’s works. Looks like kernel image and dtb included in kernel snap supports the network driver.