Trying it worked with:
First and foremost, activate extglob
(on bash, on zsh the syntax is quite similar)
shopt -s extglob
then
rm -r !(Applications|"Virtualbox VMs"|Downloads|Documents|Desktop|Public)
but names with spaces are (as always) tough. Tried also with Virtualbox\ VMs
instead the quotes. It deletes always that directory (Virtualbox VMs
).