The Edge of Chaos
by: Mark Stadtmueller, VP, Product Strategy
No, I am not writing about the mid-term elections in the United States.
But, one of the great things about working for a dynamic AI company like Lucd is the opportunity to constantly learn new things. It is better when the new thing is cool. And better yet when it strikes a chord and rings true.
Lucd research and development are getting ready to announce something amazing! I won’t spoil it, because we will be announcing it as SC18. But, keeping up with some of the research led me to this term: “The Edge of Chaos”. I had to find out.
This link talks about “The Edge of Chaos” and defines it by referring to a quote from a book titled “The Emerging Science at the Edge of Order and Chaos”. The Author, Mitchell Waldrop defines: “The edge of chaos is where life has enough stability to sustain itself and enough creativity to deserve the name of life.”
In Artificial Intelligence and in particular machine learning, this edge of chaos becomes very important. Without getting into Lucd research and development that we will be announcing, when working with data myself and doing “weekend data science”, I have seen the importance of this first hand. When performing a simple Regression against a data set in R, if your r-squared is too low, it means your input does not have enough information to describe the model. There is too much chaos. But, also, once, I got the warning “glm.fit: fitted probabilities numerically 0 or 1 occurred”. After searching stackoverflow, I learned one of my input features was completely describing the result. Basically, there was nothing to learn. The data was static. It was boring. I had to remove some structure in order to learn anything. Even though I did not know it at the time, the model learning required had to occur between having too much order and having too much chaos.
And don’t we see this in product development as well? It seems to me, Agile tries to operate in this innovative area between order and chaos.
In fact, we see this in business as well. If everything is ordered, it is stifling and the likelihood of being overtaken is high. But, if there is no structure, nothing gets done.
Really, how we organize ourselves in government has to operate here, providing both structure and freedom. Waldrop goes on to say: “The edge of chaos is where new ideas and innovative genotypes are forever nibbling away at the edges of the status quo, and where even the most entrenched old guard will eventually be overthrown. The edge of chaos is where centuries of slavery and segregation suddenly give way to the civil rights movement of the 1950s and 1960s; where seventy years of Soviet communism suddenly give way to political turmoil and ferment; where eons of evolutionary stability suddenly give way to wholesale species transformation. The edge is the constantly shifting battle zone between stagnation and anarchy, the one place where a complex system can be spontaneous, adaptive and alive.”
Math is always describing amazing things, but how cool is that!