Point to server IP address instead of using your name servers?

Yeah for several reasons I would rather use my domain provider DNS server and just point the cname to the server.

Can I do that?

Here you go: Docs - InfinityFree Forum