The Art of Scientific Investigation: How Intuition and the Imagination Fuel Scientific Discovery and Creativity
“Those who do not know the torment of the unknown cannot have the joy of discovery.”
By Maria Popova
Last week, we took in some timeless vintage wisdom on the role of serendipity and chance-opportunism in creativity and scientific discovery, culled from the 1957 gem The Art of Scientific Investigation (public library; public domain) by Cambridge University animal pathology professor W. I. B. Beveridge — a brilliant treatise on creativity in science and, by extension, in all endeavors of the mind. Beveridge constructs what’s essentially a florilegium of quotes by famous scientists and case studies of watershed discoveries to synthesize insights on what makes successful science — and successful creative thinking in general, exploring subjects like serendipity, intuition, and imagination to reveal the habits of mind that produce good ideas.
Beveridge cites philosopher, psychologist, and educational reformer John Dewey‘s seminal 1933 book, How We Think, outlining Dewey’s model for conscious thinking: First we become aware of the difficulty or problem, which provides the stimulus; then, a suggested solution pops into the conscious mind; finally, a reason evaluates the idea to reject or accept it — if the idea is rejected, the mind goes back to the previous step and repeats. Beveridge offers a brilliant articulation of the combinatorial creativity that underlies what we often call intuition:
The important thing to realize is that the conjuring up of the idea is not a deliberate, voluntary act. It is something that happens to us rather than something we do.
In ordinary thinking ideas continually ‘occur’ to us in this fashion to bridge over the steps in reasoning and we are so accustomed to the process that we are hardly aware of it. Usually the new ideas and combinations result from the immediately preceding thought calling up associations that have been developed in the mind by past experience and education.
In allowing for these magic moments to occur, Beveridge stresses the importance of embracing uncertainty and doubt:
Many people will not tolerate a state of doubt, either because they will not endure the mental discomfort of it or because they regard it as evidence of inferiority.
He once again quotes Dewey, who advocated what he called “reflective thinking”:
To be genuinely thoughtful, we must be willing to sustain and protract that state of doubt which is the stimulus to thorough enquiry, so as not to accept an idea or make a positive assertion of a belief, until justifying reasons have been found.
Further synthesizing Dewey, Beveridge captures the heart of how I, too, believe creativity works:
It is not possible deliberately to create ideas or to control their creation. When a difficulty stimulates the mind, suggested solutions just automatically spring into the consciousness. The variety and quality of the suggestions are functions of how well prepared our mind is by past experience and education pertinent to the particular problem. What we can do deliberately is to prepare our minds in this way, voluntarily direct our thoughts to a certain problem, hold attention on that problem and appraise the various suggestions thrown up by the subconscious mind. The intellectual element in thinking is, Dewey says, what we do with the suggestions after they arise.
Other things being equal, the greater our store of knowledge, the more likely it is that significant combinations will be thrown up. Furthermore, original combinations are more likely to come into being if there is available a breadth of knowledge extending into related or even distant branches of knowledge.
I frequently use LEGO as a metaphor for combinatorial creativity — if we only have bricks of one shape, size, and color, what we build with them remains limited; but if we build with pieces of various shapes, sizes, and colors, our creations will be infinitely more interesting. Beveridge corroborates this by citing Dr. E. L. Taylor:
New associations and fresh ideas are more likely to come out of a varied store of memories and experience than out of a collection that is all of one kind.
Further confirming what Einstein, Anne Lamott, and Steve Jobs have said about rationality and intuition, Beveridge cites iconic physicist Max Planck, father of quantum physics:
Again and again the imaginary plan on which one attempts to build up order breaks down and then we must try another. This imaginative vision and faith in the ultimate success are indispensable. The pure rationalist has no place here.
Indeed, Einstein himself put it thusly:
There is no logical way to the discovery of these elemental laws. There is only the way of intuition, which is helped by a feeling for the order lying behind the appearance.
Beveridge goes on to argue that intuition is really a pattern of ideas that forms as we accumulate experiences and education, and even relates this to our well-documented fear of being wrong:
The instinctive sense of irritation we feel when someone disagrees with us or when some fact arises which is contrary to our beliefs may be due to the break in the pattern we have formed.
Writing about the importance of imagination in science, the prominent 19th-century physicist John Tyndall insisted:
Newton’s passage from a falling apple to a falling moon was an act of the prepared imagination. Out of the facts of chemistry the constructive imagination of Dalton formed the atomic theory. Davy was richly endowed with the imaginative faculty, while with Faraday its exercise was incessant, preceding, accompanying and guiding all his experiments. His strength and fertility as a discoverer are to be referred in great part to the stimulus of the imagination.
Beveridge sums it up beautifully:
Facts and ideas are dead in themselves and it is the imagination that gives life to them. But dreams and speculations are idle fantasies unless reason turns them to useful purpose. Vague ideas captured on flights of fancy have to be reduced to specific propositions and hypotheses.
Echoing Carl Sagan’s wisdom on the balance between skepticism and open-mindedness, he continues:
While imagination is the source of inspiration in seeking new knowledge, it can also be dangerous if not subjected to discipline; a fertile imagination needs to be balanced by criticism and judgment. This is, of course, quite different from saying it should be repressed or crushed. The imagination merely enables us to wander into the darkness of the unknown where, by the dim light of the knowledge that we carry, we may glimpse something that seems of interest. But when we bring it out and examine it more closely it usually proves to be only trash whose glitter had caught our attention. Things not clearly seen often take on grotesque forms. Imagination is at once the source of all hope and inspiration but also of frustration. To forget this is to court despair.
Returning once again to the pivotal role of embracing failure, Beveridge writes:
The scientist who is excessively cautious is not likely to make either errors or discoveries… Humphry Davy said: ‘The most important of my discoveries have been suggested to me by my failures.’ The trained thinker shows to great advantage over the untrained person in his reaction to finding his idea to be wrong. The former profits from his mistakes as much as from his successes.
Dewey says: “What merely annoys and discourages a person not accustomed to thinking … is a stimulus and guide to the trained enquirer… It either brings to light a new problem or helps to define and clarify the problem.”
To that effect, upon receiving the Nobel Prize in physics, Max Planck remarked:
Looking back … over the long and labyrinthine path which finally led to the discovery [of the quantum theory], I am vividly reminded of Goethe’s saying that men will always be making mistakes as long as they are striving after something.
Beveridge cautions that the most important element in harnessing the power of the imagination is avoiding the trap of conditioned thinking:
Psychologists have observed that once we have made an error, as for example in adding up a column of figures, we have a tendency to repeat it again and again. This phenomenon is known as the persistent error. The same thing happens when we ponder over a problem; each time our thoughts take a certain course, the more likely is that course to be followed the next time. Associations form between the ideas in the chain of thoughts and become firmer each time they are used, until finally the connections are so well established that the chain is very difficult to break. Thinking becomes conditioned just as conditioned reflexes are formed. We may have enough data to arrive at a solution to the problem, but, once we have adopted an unprofitable line of thought, the oftener we pursue it, the harder it is for us to adopt the profitable line.
After offering several first-hand accounts of discovery by prominent scientists, Berveridge summarizes the gist of intuition:
The most characteristic circumstances of an intuition are a period of intense work on the problem accompanied by a desire for its solution, abandonment of the work perhaps with attention to something else, then the appearance of the idea with dramatic suddenness and often a sense of certainty. Often there is a feeling of exhilaration and perhaps surprise that the idea had not been thought of previously.
The psychology of the phenomenon is not thoroughly understood. There is a fairly general, though not universal, agreement that intuitions arise from the subconscious activities of the mind which has continued to turn over the problem even though perhaps consciously the mind is no longer giving it attention.
(Of course, though the exact mechanisms of ideation remain, and possibly always will, not fully understood, in the half-century since Beveridge’s work psychology and neuroscience have done a great deal to shed some light on how creativity works and what happens backstage in the brain.)
Beveridge outlines the process thusly:
Ideas spring straight into the conscious mind without our having deliberately formed them. Evidently they originate from the subconscious activities of the mind which, when directed at a problem, immediately brings together various ideas which have been associated with that particular subject before. When a possibly significant combination is found it is presented to the conscious mind for appraisal. Intuitions coming when we are consciously thinking about a problem are merely ideas that are more startling than usual. But some further explanation is needed to account for intuitions coming when our conscious mind is no longer dwelling on that subject. The subconscious mind has probably continued to be occupied with the problem and has suddenly found a significant combination. Now, a new idea arriving during conscious thinking often produces a certain emotional reaction — we feel pleased about it and perhaps somewhat excited. Perhaps the subconscious mind is also capable of reacting in this way and this has the effect of bringing the idea into the conscious mind.
Because such intuitive ideas often vanish quickly after their appearance, Beveridge recommends a “valuable device” for capturing them: the habit of carrying pencil and paper to note down original ideas. (Or, fifty years later, Evernote, my preferred alternative.) He advises:
Ideas often make their appearance in the fringe of consciousness when one is reading, writing or otherwise engaged mentally on a theme which it is not desirable to interrupt. These ideas should be roughly jotted down as quickly as possible; this not only preserves them but also serves the useful purpose of getting them ‘off your mind’ with the minimum interruption to the main interest. Concentration requires that the mind should not be distracted by retaining ideas on the fringe of consciousness.
To underline the role of the intuition, Beveridge quotes German-British philosopher F. C. S. Schiller:
It is not too much to say that the more deference men of science have paid to logic, the worse it has been for the scientific value of their reasoning… Fortunately for the world, however, the great men of science have usually been kept in salutary ignorance of the logical tradition.
Beveridge circles back to the importance of developing a comfort level with the unknown, citing the celebrated French physiologist Claude Bernard:
Those who do not know the torment of the unknown cannot have the joy of discovery.
The Art of Scientific Investigation is available as a free download in multiple formats from The Internet Archive, but be aware the text was digitized poorly using optical character recognition and is plagued with legibility errors.
Published June 1, 2012