Remote Access only working with VPN

So, I was trying to Port Forwarding and I’ve successfully connected but only works with VPN. When it’s turned off, it cannot connect but if I enable VPN it’s connected. How to solve this?