The Perils of Parallel
By Justin Lapre
Think of your mattress at home. If you bring it to the bowling alley and place bowling balls of varying weights on the mattress you should have indentations of varying depth (barring any memory foam magic!) If you take a marble and roll it towards one of these grooves, it will likely follow it downhill to the nearest bowling ball as you may expect. Typically in machine learning applications the intent is to optimize an objective function e.g., finding the deepest groove in the mattress. This brings up an often overlooked fact that rarely do optimizations find the globally optimal value but rather values that are increasingly “better.” The trouble with this approach is that the marble may get “captured” by a nearby groove in your mattress even though there may be a deeper groove elsewhere. We have a seemingly obvious solution here: for example we can use multiple CPUs or GPUs, hereinafter referred to collectively as cores, to break up our solution space and try more simultaneous locations. This will allow us to more quickly explore the parameter space to find better and better values for our objective function. But let us delve a bit deeper: do you remember Amdahl’s Law from your undergraduate computer organization class? See Figure 1 for a refresher. It states that the limit of your speedup is tied not only to the number of cores you are using to solve your problem but also to the degree of sequential (i.e., non-parallel) code in your program:
Figure 1: Amdahl’s Law implies that, in order to maximize the benefit of adding additional cores, the amount of sequential code must also be minimized.
Which brings me to my point that scalability is hard. Enabling your team to more quickly train your models or make faster decisions is critical to your business. The team at Lucd has a proven track record of delivering impressive results across clusters of up to ∼2 million cores. Our technology is a best of breed hybrid of decades of research in parallel simulation and industrial grade machine learning. Would that be something that you are interested in?