Checkout the SNAPCRAFT_BUILD_ENVIRONMENT_MEMORY environment variable:
by default it allocates 2GiB (the documentation is wrong but I currently can’t fix it)
One of the downside of a VM build is that it will always allocate all memory even if the guest OS isn’t use much. You might want to try building it in an LXD container instead.