Have you ever wondered how your connection from outside of Azure to your database is handled? It is important to understand that there is a difference between route(s) from when connecting inside to that of outside of Azure.

When outside of Azure your TCP session is established via the Azure SQL Database gateway and all subsequent packets flow via the gateway, as shown below. (proxy connections)

These gateways have IP addresses and it depends on your location. For example, my server is West Europe after a NSLOOKUP command.

As mentioned within this article (https://docs.microsoft.com/en-us/azure/azure-sql/database/connectivity-architecture) you can see this is true.

Here is another one from my list.

If you fancy going down a different route then there is a concept of private link which might be what you need where with this feature it creates a private endpoint which maps a private IP Address from the Virtual Network to your Azure SQL Database.