Point to server IP address instead of using your name servers?

Yeah for several reasons I would rather use my domain provider DNS server and just point the cname to the server.

Can I do that?

Here you go: https://infinityfree.net/support/domains/find-your-ip