Hello and welcome to the communities.
Note: Discussion successfully moved from VMware ESXi 5 to Backup & Recovery
1) Can I back up the host configuration?
You can use the esxcfg-backup command to backup the host config.
2) Can I back up the VM's?
Definitely. There are many options here.
3) Is there a one solution that can do it all provided all data is store on this machine?
Not that I know of, but it isn't a big deal really. You will want to use the esxcfg-backup command anytime you make changes to the host, which shouldn't be that often. For the VMs - what licensing do you have and what existing (if any) backup solution do you have?