Docker container running fast when connected to localhost db but very slow on external db

Question

I'm running a php application on a docker-container. When I connect to the local database responses are very fast (< 1 second). When I connect to an external db (running on google cloud or amazon aws) the performance is really slow (> 35 seconds).

I have tried using Google's DNS as described in a few links I found but no luck. My DNS resolution is very fast inside the docker container and to connect to Google Cloud SQL DB I'm using the IP address directly.

I have checked this link web server running inside a docker container running inside an EC2 instance responses very slowly and this Docker slow non-local database access. It seems to be something related, but not sure.

I think it's a Docker problem, or somewhat related to the container because the same remote db (on google cloud and aws) is used on other applications and the speed is very fast. In my opinion it is something related to the network inside the container.

So, to sum it up here are the scenarios I used for testing (DB contents are the exact same):

1) On My Mac as Localhost running my app inside a Docker container:

• DB on my localhost (MAMP): very fast (< 1 second);
• DB on Google Cloud SQL: very slow (> 35 seconds);
• DB on Amazon RDS: very slow (> 35 seconds);

2) On a Google Compute Engine with my app running inside a Docker container:

• DB on Google Cloud SQL: very slow (> 35 seconds);
• DB on Amazon RDS: very slow (> 35 seconds);

3) On a custom Google app engine flex environment with my app running inside Docker:

• DB on Google Cloud SQL: very slow (> 35 seconds);
• DB on Amazon RDS: very slow (> 35 seconds);

4) On a PHP Google app engine flex environment:

• DB on Google Cloud SQL: very slow (> 35 seconds);
• DB on Amazon RDS: very slow (> 35 seconds);

5) With my App running outside of Docker on a Google Compute Engine instance (PHP + apache):

• DB on Google Cloud SQL: very fast (< 1 second);
• DB on Amazon RDS: very fast (< 1 second);

6) With my App running outside of Docker on localhost (Mac):

• DB on Google Cloud SQL: very fast (< 1 second);
• DB on Amazon RDS: very fast (< 1 second);
• DB on my localhost (MAMP): very fast (< 1 second);

Does anyone know an approach to address or to find the problem? I understand this is a problem that might be hard to solve. So, my question is more related to how I should debug that to find the problem.

My Dockerfile:

    FROM php:7.0.17-apache

    RUN apt-get update
    RUN apt-get install -y apt-utils curl vim
    RUN docker-php-ext-install mysqli && docker-php-ext-enable mysqli
    RUN docker-php-ext-install pdo pdo_mysql && docker-php-ext-enable pdo_mysql
    RUN pecl install xdebug
    # The base image does not have php.ini.
    # Copy our own, with xdebug settings
    ADD ./php.ini /usr/local/etc/php/

    # Configure apache
    RUN a2enmod rewrite
    RUN a2dissite 000-default.conf

    # Copy sites available
    ADD ./www.metalar.net.conf /etc/apache2/sites-available/

    # Copy Ports file
    ADD ./ports.conf /etc/apache2/

    # Copy Ports file
    ADD ./apache2.conf /etc/apache2/apache2.conf

    # Copy error log
    ADD ./error.log /var/log/apache2/error.log

    # Make directory to host project files
    RUN mkdir -p /srv/www/www.metalar.net

    # Copy App to proper destination
    ADD . /srv/www/www.metalar.net

    # Enable config
    RUN a2ensite www.metalar.net.conf

    EXPOSE 8080

netstat -s

Ip:
    187 total packets received
    0 forwarded
    0 incoming packets discarded
    187 incoming packets delivered
    163 requests sent out
Icmp:
    0 ICMP messages received
    0 input ICMP message failed.
    ICMP input histogram:
    0 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
Tcp:
    2 active connections openings
    0 passive connection openings
    0 failed connection attempts
    0 connection resets received
    0 connections established
    181 segments received
    157 segments send out
    0 segments retransmited
    0 bad segments received.
    0 resets sent
Udp:
    6 packets received
    0 packets to unknown port received.
    0 packet receive errors
    6 packets sent
UdpLite:
TcpExt:
    2 TCP sockets finished time wait in fast timer
    171 packet headers predicted
    4 acknowledgments not containing data payload received
    TCPRcvCoalesce: 82
    TCPOrigDataSent: 4
IpExt:
    InOctets: 234466
    OutOctets: 7205
    InNoECTPkts: 187

Answer 1

First of all which is the version of Docker you are using?

Because some time ago I experienced a similar problem with my Docker running in the Mac and I found a known issue for Docker for MAC that could be related to you as well.

• "Experiencing sluggish I/O, particularly on MySQL official container, from container initialisation to CREATE TABLE and INSERT queries."

This issue has been mapped into the following one.

• Severe Docker 1.12.1 performance regression with DB2 images (~10x slower)

They provided a workaround, but right now should be enough to update Docker to solve the poor performances.

However these known issues would not explain behaviour #2 and #3 so I guess it is not related, but it worth to me mentioned.

Debug

The point is that is not an issue with the server or with the application, therefore I would focus on debugging the network part of Docker that for some reason is not working as expected.

These are merely general advises but you can find more info for example here or around the Docker forums.

In order to debug I would open two terminals and with the first one I would ssh into my container running all the commands in parallel with the local machine in order to compare the results.

• I would check if the speed of the network from the container is similar downloading file stored in Google cloud Storage and any random website

• Ping the MySql instance and compare the latency.

• Traceroute towards the MySql instance and check the path of the packets and the latency at each hop.

• Try to connect to the MySql through the command line and compare the timing to set up connection and perform basic operations (create, update, upload, drop, ecc).

• Through tcpdump (only from the container) I would inspect the traffic between the two instances looking for any kind or error, connection reset, packet malformation while connecting to the db instance and while running the application

• Try as well to perform some DNS lookup and compare the timing

• Inspect as well any kind of log that is present in the container side under /var/log and if you have access to the MySql instaces log

• Try to understand which are the operation that actually takes longer to be performed inside the container (is it just connecting to the server? is it the whole process that is slowed down? Is it during huge data transfers?)

• I would try to built an essential example showing the performances degradation and I would open an issue in docker corresponding GitHub repository.

If you spot any discrepancy between the commands run in localhost and in the container please update the question to see if someone could help you to actually solve the issue.

P.S. You can find also this interesting, basically a guy having issues to connect to a MySql instance and successfully debugging it