This year Stuart Kauffman et al wrote a fundamental paper "No entailing law, but enablement in the evolution of the biosphere" (you can see him speak about this here http://necsi.edu/video/
This was a great revelation to me and I think provides a deep appreciation of the constraints of a physics 'worldview' for understanding living systems. Kauffman has spoken about the 'adjacent possible' in his other writings - but here he made more concrete for me.
To be brief - Kauffman points out that evolutionary systems cannot be positioned within a 'pre-stateable phase space' that is it is impossible to pre-state all 'adjacent possibles' of a given situation. Another way of saying this is that it is only after the fact that 'Darwinian pre-adaptations' or potential affordances become visible. Kauffman give the example of the impossibility of prestating all the possible uses that a screw-driver can be put to.
In the physics worldview we posit a prestated phase space within which we can conduct a calculus of possible trajectories within that space. This is the basis of Maxwell's Demon - the imagination of an intelligence that can be aware of all the current bits (and their states) with the consequence that the future could then be simply foreseen as the inevitable computation of trajectories of the existing bits. Kauffman gives another example of evolution selecting a fish with a swim bladder. This process fits well with developments in the physics pre-stated space as the causal webs that shape the evolving fish-with-swim-bladder-in-environment event. However, when a micro-organism inhabits the swim-bladder turning it into a 'niche' - this was not prestateable, it was a-causal in the swim-bladder was not 'selected' for to be a niche. However, once existant - becoming a niche was an 'adjacent possible' that enabled a micro-organism to actualize an affordance (as adjacent possible). Kauffman says that these sorts of processes cannot be algorithmized. There is no way to develop an 'entailing law' to predict the emergence of affordances, or actualizations of adjacent possibles or make visible pre-adaptations to future contexts.
This also has salience to the difference between information (especially in relation to Shannon) versus knowledge - as embodied in socially-based minds. What Shannon was able to formulate was how to ensure a transmission of a pattern of differences (e.g. information - as difference that makes a difference) - through various transformations relevant to particular mediums through which transmissions occur, with fidelity. For example I can ensure that the ink-blot I create on a paper, scan with a scanner, attach to an email, - is received via email and printed out will be exactly like the ink-blot that I sent. Where knowledge comes is that I can't assure that the meaning I see in the ink-blot (a spider) is the meaning that the receiver sees in the ink-blot (butterfly). Shannon admitted the the theory of information did not address the 'meaning' of the 'signal'. This is what makes knowledge 'transfer' much more like translation where it is inevitable that 'information' (as meaning) is gained and lost that is outside of the control of the sender of the signal.
Thus as Kauffman notes - life creates the conditions of its own becoming. Each step in a complex living system changes the conditions (potential adjacent possibles, affordances, pre-adaptations) of the next step.
What makes 'knowledge' possible is that a tacit collective knowledge arise through the complex social transactions of synchronization - as Bateson claimed - the fundamental of mammalian communication is the harmonization of nervous systems - which I would now understand as an environment of the social minds that constitute living systems.
The puzzle is, How does an 'enablement' become seizable? For example, did the micro-organism accidentally get swallowed into the swim-bladder - a random act, and despite this random act somehow survived, found protection and learned to replicate in this new environment (producing excrement thus enabling other 'niche opportunities'). Is it possible to derive a probability for this random act before it happened? Kauffman say no. So the micro-organism doesn't receive a signal (I'm in a new niche) but rather perceives a meaning I think this is like a niche?
It is for all of the reasons above, modern economics (built on the concepts and mathematics of 19th century physics) requires a 'rational actor' - a predictable particle - rather than a meaning-making, affordance-realizing consciousness.
It is also for the above reasons that a multi-disciplinary effort is needed to continually iterate simulations related to living systems to make imaginatively useful - even if they may never create viable predictions beyond a relatively short horizon.
It has been noted that Nature/Reality is not a theory. While I wholeheartedly agree that nature is not a theory - I also agree with Einstein and vast debate in philosophy that essentially supports him - Whether you can observe a thing or not depends on the theory which you use. It is the theory which decides what can be observed.
Like Khun noted, anomalies only become evident because they challenge inadequate theory. Thus we are unable to perceive an 'unfiltered' world a world free of theory (conscious or not). Our only choice is choosing/creating theory, I think observation (in a scientific sense) is impossible without theory.
It is true that Kauffman's adjacent possibles is abstract, and very difficult to 'grasp' - attributing reality to something that is ultimately intangible - becoming tangible only after a field of potential collapses on a single possible. This surely can seem like magical or divine spontaneous creation.
Turning back to nature enables us to search more closely for those anomalies and perhaps find a better pattern that we can describe through mathematical and other types of metaphors.
I remember reading Gregory Bateson in the mid 80s. He went on at length about information being a difference that made a difference, etc. and at one point he spoke of this as a sort of difference that makes a difference - a pattern. He spoke about two types of epistemology (how we know we know), an epistemology of energy (which is the foundation of physics - the billiard balls on a pool table descriptions of the universe) and an epistemology of pattern. This later type of epistemology is the foundation of the phenomena we could call 'meaning'. He spoke about how pattern is essentially not just intangible but has literally no substance. For example the telephone call not received (nothing) 'causes' consequences. In essence where does 'difference' actually live?
It is only in the last five years, have I come to really understand this, as I was thinking through the problem of the difference between information and knowledge. What made it clear for me was thinking about Shannon's ideas of information using Bateson's metaphor of a blind man walking with a cane.
The side walk has discernible changes (pattern), using a cane and tapping the blind man is able to note important changes (pattern) in the side walk by tapping - which provides a way to transform the changes in the side walk into feeling and sound. The feeling (vibrations through the cane) and the sound (vibration of the cane-side-walk interaction through the air) are transformed again in the neural structure to produce a 'picture' that is a pattern of the side walk (I exert poetic license just to keep it simple).
Another example is how we think of something to say, that pattern of thought is transformed into a pattern of sounds (according to the structure of language), those sounds are transformed into a pattern of electro-magnetic waves to transmit via a digital medium-phone/text, then the pattern of digital differences is transformed into a pattern of language-sounds (voice), which is then transformed into a pattern of meanings (which may entail iterated exchanges to negotiate concurrence on that meaning).
In both of these examples a material medium is absolutely necessary to 'embody' the pattern, but it is only because the 'pattern' itself is not material that it is possible for the patterns to 'embody' various media through appropriate transformations for corresponding material media.
In this way differences that make a difference (pattern) is non-material, non-energetic but can only be perceived via a substantive form of media.
It could be that pattern (information) is the source of 'Formal Causation' which would be why effects as 'embodiments' would precede their cause (unknowable prior to its effects). If this has any truth it would then make certain dimensions of reality unobservable other than because we can formulate theories concerning prior effects to un-instantiated formal causes.
Part of what has got me thinking about this was a comments by a colleague related to the physics of continuity and topology.
In one of my first philosophy classes my professor was an ancient (Leslie Armour) who had been teaching for 50 years - it was like Kant and Hegel were his neighbours. :) The class was Infinity, Ethics and Society (or something like that). I forget the discussion - but I asked him about the issue of change and continuity - how could I, an obviously different person than I was when I was five - still be the 'same' person? His answer was that we probably constructed our identity through some form of narrative continuity - continuity of pattern.
This is the core of the difference between 'mere' :) physics-based systems and complex or living systems. Complex/living systems exist 'far-from-equilibrium'. Their 'integrity' or construction is maintained via a 'strange attractor' for complex systems and via a mechanism of homeostasis in living systems. One could argue that the 'identity' of a living system is the complex, dynamic set of values that homeostasis uses to regulate the system's state of 'far-from-equilibrium'. That means that unlike equilibrium (which is a sort-of path of least resistance), homeostasis is a dynamic ongoing 'regulatory-intervention' requiring energy (and enabling the generation of energy to keep the system going).
So a complex and/or living system is definitely dependant on physics - but it's identity (as continuity of pattern) may not be 'apprehendable' from a strictly physics foundation.
Another example. Human brains all have language processing structures. But there is no 'english' gene. Nor could we determine which language is being spoken only by a strict analysis of neuronal-chemical events. A particular language (patterns) arises as an emergent via complex social interactions - and once language emerges it has the capability to interate influences upon the bio-neuro-chemical substrate that is its 'medium'.
It may be that pattern (like the adjacent possibles) is an intangible non-material norenergetic domain/dimension of reality (Formal Cause), that requires material media to exist and act but whose laws can't be reduced to those laws of the particular tangible reality (medium) that embody the pattern at any particular moment.
This would not be a mysticism or a divine source of being - but a domain of reality that may require a different type of language than what mathematics currently offer.
This line of thought is an engagement that is similar to how people from different culture have to construct a third language in order to make bridges to their native foundations. :)
Maybe the challenge of a 21st century mathematics and logic is one that will seem initially to be a development of a more abstract theory - maybe to an extreme.
In my foresight work, I've often used the experience of Xerox as a metaphor - Xerox funded Xerox PARC (Palo-Alto Research Centre) where their brilliant scientist developed the desktop computer, with a GUI interface and a mouse. When executive looked at the result - they pondered then asked what that had to do with photocopies? The rest is history - the scientist showed their work to Steve Jobs and Bill Gates.
But had the executives of Xerox, taken a moment and abstracted what business Xerox was really in (its identity), they could have realized that the ultimate business of Xerox was 'information processing' and that would have enabled (in Kauffman's sense of the word) to see a different (but continuous) future (identity) of Xerox. In pondering the lessons not learned by Xerox - I think what comes after the Post-modern age is the era of the 'Meta-real' - where we must all struggle to 'abstract' a higher order sense of identity (continuity of narrative) in order to grasp the opportunities that a culturally-based evolution enables.