{"id":1425,"date":"2012-01-14T10:04:32","date_gmt":"2012-01-14T09:04:32","guid":{"rendered":"http:\/\/www.blaess.fr\/christophe\/?p=1425"},"modified":"2012-01-14T10:04:32","modified_gmt":"2012-01-14T09:04:32","slug":"parallelizing-compilations","status":"publish","type":"post","link":"https:\/\/www.blaess.fr\/christophe\/2012\/01\/14\/parallelizing-compilations\/","title":{"rendered":"Parallelizing Compilations"},"content":{"rendered":"

(Version originale en fran\u00e7ais<\/a>)<\/p>\n

I very frequently compile Linux kernels, often during training sessions or engineering services (mainly in the field of embedded systems or drivers development), sometimes while writing articles or books.<\/p>\n

Compilation time varies greatly depending on the amount of code (drivers, filesystems, protocols, etc.) and on the CPU power of the host machine. On a mid-range PC, compiling a kernel adjusted for an embedded system (with very few drivers) lasts about three minutes. On an entry level machine (or a little old one), compiling a generic kernel for PC (with hundreds of drivers as modules) can last an hour.<\/p>\n

\n\n<\/p>\n

To take advantage of the parallelism offered by the current processors (multiprocessor, multicore or hyper-threading), the make<\/code> command allows us to run multiple jobs. So<\/p>\n

$ make -j 4<\/strong><\/pre>\n
guarantees there is always four compilation jobs active.<\/p>\n
I have long reiterated that \u00ab\u00a0if you have N processors (or cores, or virtual CPUs) available, you will save time by starting 2N compilation jobs in parallel<\/em>\u00ab\u00a0. The idea is that for every proccessor we have a job that performs the compilation (consuming CPU time) while another job is saving the results of the previous compilation and loading the source file of the next task. But wait… is this true?<\/p>\n
Test script<\/h1>\n
I wrote the following script which downloads and unpack the required kernel sources, then makes several compilations using a variable number of jobs. For example, if we start<\/p>\n
$ .\/test-make-j.sh 3 5 8<\/strong><\/pre>\n
It performs three complete compilations: one with three tasks in parallel, one with the five jobs and the last with eight jobs. The results are recorded in a text file. The script follows.<\/p>\n
test-make-j.sh <\/strong><\/a>\n#! \/bin\/sh\n\nKERNEL_VERSION=\"linux-3.2\"\nKERNEL_URL_PATH=\"www.kernel.org\/pub\/linux\/kernel\/v3.0\/\"\nRESULT_FILE=\"compilation-timing.txt\"\n\nif [ \"$#\" -eq 0 ]\nthen\n  echo \"usage: $@ jobs_number...\" >& 2\n  exit 0\nfi\n\nif [ ! -d \"${KERNEL_VERSION}\" ]\nthen\n  if [ ! -f \"${KERNEL_VERSION}.tar.bz2\" ]\n  then\n    wget \"${KERNEL_URL_PATH}\/${KERNEL_VERSION}.tar.bz2\"\n    if [ $? -ne 0 ] || [ ! -f \"${KERNEL_VERSION}.tar.bz2\" ]\n    then\n      echo \"unable to obtain ${KERNEL_VERSION} archive\" >&2\n      exit 1\n    fi\n  fi\n  tar xjf \"${KERNEL_VERSION}.tar.bz2\"\n  if [ $? -ne 0 ]\n  then\n    echo \"Error while uncompressing kernel archive\" >&2\n    exit 1\n  fi\nfi\n\ncd \"${KERNEL_VERSION}\"\n\necho \"# Timings of ${KERNEL_VERSION} compilations\" >> \"${RESULT_FILE}\"\nnb_cpu=$(grep \"^processor\" \/proc\/cpuinfo | wc -l)\n\necho \"# Processors: ${nb_cpu}\" >> \"${RESULT_FILE}\"\naffinity=$(taskset -p $$ | sed -e 's\/^.*:\/\/') >> \"${RESULT_FILE}\"\n\necho \"# Affinity mask: ${affinity}\" >> \"${RESULT_FILE}\"\nfor nb in \"$@\"\ndo\n  echo \"# Compiling with $nb simultaneous jobs\" >> \"${RESULT_FILE}\"\n  make mrproper<\/strong>\n  make i386_defconfig<\/strong>\n  sync\n  sleep 10 # Let's all calm down\n  start=$(date \"+%s\")\n  make -j $nb<\/strong>\n  sync\n  end=$(date \"+%s\")\n  # This script will fail during february 2038 ;-)\n  echo \"$nb     $((end - start))\" >> \"${RESULT_FILE}\"\ndone<\/pre>\n
Results<\/h1>\n
Here are the results of a run on an Intel Q6600 Quad-Core (file: Intel-Q6600-1.txt<\/a>)<\/p>\n
# Timings of linux-3.2 compilations\n# Processors: 4\n# Affinity mask:  f\n# Compiling with 1 simultaneous jobs\n1     675\n# Compiling with 2 simultaneous jobs\n2     346\n# Compiling with 3 simultaneous jobs\n3     241\n# Compiling with 4 simultaneous jobs\n4     197\n# Compiling with 5 simultaneous jobs\n5     198\n# Compiling with 6 simultaneous jobs\n6     194\n# Compiling with 7 simultaneous jobs\n7     195\n# Compiling with 8 simultaneous jobs\n8     196\n# Compiling with 9 simultaneous jobs\n9     197\n# Compiling with 10 simultaneous jobs\n10     198\n# Compiling with 11 simultaneous jobs\n11     198\n# Compiling with 12 simultaneous jobs\n12     198\n# Compiling with 13 simultaneous jobs\n13     200\n# Compiling with 14 simultaneous jobs\n14     201\n# Compiling with 15 simultaneous jobs\n15     201\n# Compiling with 16 simultaneous jobs\n16     200<\/pre>\n
Let’s see them graphically with this little Gnuplot command line. On the horizontal axis, lies the number of concurrent jobs and on the vertical axis is the compilation time (in seconds).<\/p>\n
$ echo \"set terminal png size 640,480 ; set output '.\/Intel-Q6600-1.png'; plot 'Intel-Q6600-1.txt' with linespoints\" | gnuplot<\/strong><\/pre>\n
 <\/p>\n
\"Parallel<\/a>
Parallel Compilations on 4 CPU<\/p><\/div>\n
Apparently, the best results are achieved (with some fluctuations) with make-j 4<\/code><\/strong>. Let’s try to confirm this. Before running again the script, we limit it to two processors with the following command which binds on processors 2 and 3 all the jobs launched from the running shell.<\/p>\n
$ taskset -pc 2-3 $$<\/strong><\/pre>\n
Here are the results (file: Intel-QL6600-2.txt<\/a>).<\/p>\n
# Timings of linux-3.2 compilations\n# Processors: 4\n# Affinity mask:  c\n# Compiling with 1 simultaneous jobs\n1     684\n# Compiling with 2 simultaneous jobs\n2     360\n# Compiling with 3 simultaneous jobs\n3     362\n# Compiling with 4 simultaneous jobs\n4     366\n# Compiling with 8 simultaneous jobs\n8     370\n# Compiling with 16 simultaneous jobs\n16     376\n# Compiling with 32 simultaneous jobs\n32     377\n# Compiling with 64 simultaneous jobs\n64     378<\/pre>\n
\"Parallel<\/a>
Parallel Compilations on 2 CPU<\/p><\/div>\n
This time, it is clear that the minimum time is achieved with make-j 2<\/code><\/strong>. If we repeat the experiment on a single CPU, we obtain the following values (file: Intel-Q6600-3.txt<\/a>).<\/p>\n
# Timings of linux-3.2 compilations\n# Processors: 4\n# Affinity mask:  8\n# Compiling with 1 simultaneous jobs\n1     683\n# Compiling with 2 simultaneous jobs\n2     698\n# Compiling with 3 simultaneous jobs\n3     708\n# Compiling with 4 simultaneous jobs\n4     709\n# Compiling with 5 simultaneous jobs\n5     719\n# Compiling with 6 simultaneous jobs\n6     719\n# Compiling with 7 simultaneous jobs\n7     720\n# Compiling with 8 simultaneous jobs\n8     724<\/pre>\n
Represented on the graph below:<\/p>\n
\"Parallel<\/a>
Parallel Compilations on a single CPU<\/p><\/div>\n
We can group these three curves on a single graph to better see their scales (I have not extended the curve of the compilation on a single CPU, but we can imagine that it continues with a slight increase).<\/p>\n
\"Parallel<\/a>
Parallel Compilations on Q6600<\/p><\/div>\n
To be sure, we can repeat the experiment on a different processor with two cores (AMD QL66). The results are as follows (file: AMD-QL66-1.txt<\/a>).<\/p>\n
# Timings of linux-3.2 compilations\n# Processors: 2\n# Affinity mask:  3\n# Compiling with 1 simultaneous jobs\n1     1113\n# Compiling with 2 simultaneous jobs\n2     844\n# Compiling with 3 simultaneous jobs\n3     875\n# Compiling with 4 simultaneous jobs\n4     863\n# Compiling with 5 simultaneous jobs\n5     840\n# Compiling with 6 simultaneous jobs\n6     844\n# Compiling with 7 simultaneous jobs\n7     844\n# Compiling with 8 simultaneous jobs\n8     851<\/pre>\n
\"Parallel<\/a>
Parallel Compilations on two CPU<\/p><\/div>\n
Let’s try one last experiment on the same machine (two CPU), by disabling two elements:<\/p>\n