I am using Docker to isolate a specific process. This process is run repeatedly a number of times on a multi-core virtual machine.
Each execution time is measured by its wall clock time and recorded. I'm looking to get time differences smaller than 200 ms. Unfortunately, I get about 1 second difference between the best and worst executions in Docker. I don't understand why. I want to bring it down to < 200 ms.
Here is a chart to illustrate my problem:
Here the blue columns represent the native time executions in ms, which are pretty consistent, while the orange columns show the execution times when the same code is run as a Docker process.
My goals is to get consistent execution times in Docker.
Here is my minimal reproducible example:
mem.cpp This program performs memory expensive operations to take time.
#include <bits/stdc++.h>
#include <vector>
using namespace std;
string CustomString(int len)
{
string result = "";
for (int i = 0; i<len; i++)
result = result + 'm';
return result;
}
int main()
{
int len = 320;
std::vector< string > arr;
for (int i = 0; i < 100000; i++) {
string s = CustomString(len);
arr.push_back(s);
}
cout<<arr[10] <<"\n";
return 0;
}
script.sh This script is the starting point for the Docker containers and it compiles and runs the above C++ program and records its wall time.
#!/bin/bash
# compile the file
g++ -O2 -std=c++17 -Wall -o _sol mem.cpp
# execute file and record execution time (wall clock)
ts=$(date +%s%N)
./_sol
echo $((($(date +%s%N) - $ts)/1000000)) ms
python program. It uses ProcessPoolExecutor for parallelism. It copies the files into the Docker containers and executes script.sh.
import docker
import logging
import os
import tarfile
import tempfile
from concurrent.futures import ProcessPoolExecutor
log_format = '%(asctime)s %(threadName)s %(levelname)s: %(message)s'
dkr = docker.from_env()
def task():
ctr = dkr.containers.create("gcc:12-bullseye", command="/home/script.sh", working_dir="/home")
# copy files into container
cp_to_container(ctr, "./mem.cpp", "/home/mem.cpp")
cp_to_container(ctr, "./script.sh", "/home/script.sh")
# run container and capture logs
ctr.start()
ec = ctr.wait()
logs = ctr.logs().decode()
ctr.stop()
ctr.remove()
# handle error
if (code := ec['StatusCode']) != 0:
logging.error(f"Error occurred during execution with exit code {code}")
logging.info(logs)
def file_to_tar(src: str, fname: str):
f = tempfile.NamedTemporaryFile()
abs_src = os.path.abspath(src)
with tarfile.open(fileobj=f, mode='w') as tar:
tar.add(abs_src, arcname=fname, recursive=False)
f.seek(0)
return f
def cp_to_container(ctr, src: str, dst: str):
(dir, fname) = os.path.split(os.path.abspath(dst))
with file_to_tar(src, fname) as tar:
ctr.put_archive(dir, tar)
if __name__ == "__main__":
# set logging level
logging.basicConfig(level=logging.INFO, format=log_format)
# start ProcessPoolExecutor
ppex = ProcessPoolExecutor(max_workers=max(os.cpu_count()-1,1))
for _ in range(21):
ppex.submit(task)
I have tried to use much fewer of the available CPU cores (4 or less out of 8) to make sure that the OS can utilize 4 or more for its own purposes, but that doesn't help. This makes me think the reason lies within Docker Engine most likely.
EDIT:
I tried using the newly released gcc:13-bookworm image and it performs better than native and much better than gcc:12-bullseye. Also, the times are a lot more consistent. This makes me think it has to do something with the image?
