First past the post

Contents

• Introduction
• Sections
• Projections
• A cubical world
• From objects to the eye
• The real world
• Conclusions
• References

Introduction

Figure 1

We live in a three dimensional world, which for present purposes might be thought of as a collection of simple coloured shapes floating about in a cube. Shapes along the lines of those that you can create in two dimensions in Microsoft’s Powerpoint product. Perhaps something like Figure 1 above. 

The present interest is in how the eyes and brain work together to produce the subjective experience of such a world. How much of that experience is captured in purely visual terms and how much is somewhere else. And given that vision is rather two dimensional, the various ways in which this three dimensional world might be reduced to two dimensions.

The one dimensional option was considered at references 8 and 9 – one advantage of this last being that it would be cheap; cheap in neurons and cheap in energy. These posts also introduced Figure 1 above.

We note, but do not address, the problem that the brain very probably takes the inbound image apart, before somehow putting it together again for projection into the outbound subjective experience.

We note, but do not address, the fact that rectilinear objects like cubes are far more readily computed when they are viewed from an angle than when they are view face on. Also that rectilinear objects are relatively rare in the wild, and rounded objects, like most animals, do not present the same difficulty, particularly given that when actually viewed face on, there are the very distinctive eyes.

A rather different world to that of the post at reference 4, some of the conclusions of which make their way into the present post. Different in the sense that the four objects above are simple, idealised geometric shapes, the sort of thing that is mostly to be found among the coloured wooden bricks which children used to play with when I was young. Not Brueghel’s more or less real people doing real stuff at all.

We mainly restrict our attention to the shape of things and to their arrangement with respect to each other in space, rather than, for example, to the subtlety of their colours and textures, let alone their interiors. Or put another way to their direction, altitude and range; the sort of numbers a radar set might give for an aeroplane.

A restriction which chimes with our day to day needs in the world: to cross the road without hitting the bus, to reach the glass down from the shelf, to hide away from the tiger lurking in the long grass. A world containing persistent and distinctive objects, objects with which we need to interact – or, at the very least, avoid. With the diameter of most of these objects being in the range 1cm to 10m, that is to say three orders of magnitude. The world which our sophisticated eyes and brain have equipped us to see and comprehend. 

Figure 2

These rigid, opaque, geometric shapes, might be moving about, but they do not change shape. We start with objects floating around in the void, before moving onto to the more theatrical world suggested by Figure 2. A world with a stage, wings and backdrop. A world in which our foreground objects might be labelled either with common nouns like cone and cube or with proper nouns like Carol and Carlotta, labels which might make it to consciousness when we attend to the corresponding object. A world where the point of view is much more in evidence. A world not so far removed in visual terms from the forests and savannahs in which we started out, say half a million years ago.

Note that this particular stage has a tendency to pop-out rather than stay in: an artist would probably know why. While the originally pink background to the objects was removed by Powerpoint’s useful ‘remove background’ tool. Television people must do something similar when they want to give talking heads a more interesting background than they can really afford or to put clever graphics behind the weather men and women.

Sections

Figure 3

Taking a section is a well tried way of getting from three dimensions to two, as illustrated in the figure above, a more than hundred year old geological section. What you would see if you were able to dig a huge trench across miles of countryside, with quarries being the nearest we actually get. One can occasionally see something of the sort when they dig trenches for water or other utilities here at Epsom, in the form of thin bands of chalk, the chalk of the neighbouring downs sinking down under the clay of the Thames river basin. 

Figure 4

One can also see rather weathered versions in cliffs, cliffs at the seaside or cliffs on rivers. For example the cliffs at the two ends of the spine running west to east through Isle of Wight, from Alum Bay to Yaverland beach. With the strata of the former (above) being more or less vertical, but rather less so by the time they get to Yaverland. And I note in passing that some of the oldest fossils in the world are to be found in the river cliffs of Siberia.

Nevertheless, sections of this sort are mostly an exercise of the imagination, an exercise of science: not something that one can do for oneself, for real. As is the business of imagining what these strata get up to as they move inland.

Figure 5

While the geometrical simplicity of the geological section is rather corrupted by the intrusion of three dimensional elements in the section of a kidney above left, perhaps better described as a sectional diagram rather than a section proper. A diagram which is a good deal more informative than what you get from the real section top right or the whole thing bottom right – this last being what you might actually see. More informative to a lay person at least. 

And this sort of section is real in the sense that one can actually make a section of this sort, provided that is that the kidney’s original owner has no further use for it.

Figure 6

A horizontal section through the middle of the first figure, Figure 1, might produce something like the left hand panel of Figure 6 above. A section which the brain would be well able to produce from a not very realistic scene like this one as it does have all the necessary information, but a section which it does not project into the subjective experience – although it can go so far as to direct a forearm to draw such a section on a piece of paper. Experiment suggests that when simple objects like these are vertical, drawing the horizontal section is easy enough. Making proper allowance for tilted objects is more challenging, although practise would no doubt make perfect.

While in the right hand panel we add a couple of possibilities for one dimensional sections, showing that sectioning is a technique which extends readily enough to one dimension, even though the results will not always be very informative. One needs to have a care when deciding from where to take the sections. Or, put another way, one needs to know quite a lot about what is happening in three dimensions before one can make a useful reduction to two, let alone one.

In which connection, note that there may be discontinuities, even with such simple objects as those above. If we move the sloping section gently to the left, the yellow segment will gently shrink and finally vanish. Whereas if we move it gently to the right, the green segment will stay the same and then abruptly vanish.

Note also that it may not be possible to take a horizontal section in which all four objects appear, although a middle section sloping gently up from left to right might do. A problem which some diagram drawers mitigate by suggesting the position of missing objects with dotted outlines.

So one problem is that the section may vary, possibly wildly, with the position of the section. Is it really capturing what we want to know, what we need to know about the bit of the world being sectioned? Another, more serious, is that we do not see sections: we see things from the outside, not from the inside – leaving aside perverse cases like Jonah being inside the stomach of his whale.

And, for most people, most of the time, getting a decent visual experience depends on getting plenty of input from the eyes. The brain might well modify what is coming in, but it seems to need that input to power up the subjective visual experience. Maybe riding a horse is the right analogy: you might be able to steer the horse, you might be more or less in charge, but if you want to go for a ride, you do need the horse in the first place.

We leave aside the visual experiences that some people, some of the time, are able to generate in the absence of input from the eyes, possibly even in spite of such input. We have heard, for example, of people who go blind in later life experiencing very vivid visual hallucinations, although in the small sample involved in the work reported at reference 10 these hallucinations might be vivid, but they are a long way from what a sighted person might see.

Projections

Projection, rather more technical and fiddly than a section, is, nevertheless, another widely used reductive technique, that is to say technique from getting from three dimensions to two. In the beginning, used for getting from the curved surface of the earth, more or less a sphere, to a map or to the flat pages of an atlas, a matter which became important towards the end of the fifteenth century, that is to say more than six hundred years ago.

Figure 7

The sort of projection we first had in mind here was indeed the sort of thing that one does to put the surface of the earth onto a flat piece of paper. There are lots of ways of doing this, with the well known (and attractively simple) Mercator projection shown above right being widely used. Some of these projections can be visualised by wrapping a piece of paper around a translucent globe, illuminated from within. The map on the surface of the globe is then projected onto the piece of paper and fixed in some way. The piece of paper can then be unrolled and one has one’s projection on it. All this is illustrated in the figure above, lifted from the Encyclopedia Britannica, complete with the massive distortions on the right at the higher latitudes, top and bottom. A more serious explanation is given at reference 1.

Mercator is not terribly helpful in the present context, but there is another sort of projection which might help visualise an approximation to what gets onto the retinas of the eye.

Figure 8

Not really a back projection, for which see reference 2, but the term is suggestive. We imagine a flat screen between the objects in view and the eye, with the thick blue, red and green lines suggesting what might get onto the screen. A colour determined by the first past the post rule of our title; that is to say the colour of the first object that a ray leaving the eye hits, if any. Leaving aside the complication that rays of light are actually travelling from the objects to the eye, not the other way around. Leaving aside the complication that the colour of the objects we see is as much a function of the ambient light as of the objects themselves.

A figure which generalises easily enough to three dimensional objects and a two dimensional screen, which last can be transferred without further ado to the printed page or to the screen of a computer.

Figure 9

Turning it round the other way, to make it something more like back projection doesn’t really work. What one would get in real life is the shadow picture of option 1 above. By fiddling around with transparent panels, one might get option 2, with colour, but an option which runs into trouble when objects start to occlude each other. An option which we might describe as last past the post, with the colour on the screen being the last colour a ray touches, rather than the first. But see reference 3 for an oriental take on this sort of thing.

In any event, it is clear that vision in a world of mostly opaque objects, depending as it does on the first past the post arrangement of Figure 8 above, starts with a two dimensional signal from the world. A signal which passes through the two dimensional retinas. To be processed in successive areas of the visual pathways in the brain. With the cortex of the brain itself being another two dimensional structure, millimetres thick but, taking the two hemispheres together, a quarter of a square metre in area. With the argument both here and at reference 5 being that a small, two dimensional sheet of cortex, organised into layers and mostly topically organised, is the jumping off point for the subjective experience of vision.

A cubical world

Suppose now that the world of Figure 1 is indeed a cube, a sort of cubical glass box, which we are looking at from the outside. A box with six sides: front, back, left side, right side, top and bottom. Each of these sides corresponds to a point of view, with one point of view apt to be very different to another.

Figure 1 is a sophisticated reduction of the scene to two dimensions, as viewed from the front, with the objects seeming to be lit from the behind the viewer, shadow of viewer omitted. For present purposes, we will suppose that the left hand object is blue, the right hand object is red and the two objects in the middle, the smaller cube and the cylinder, are yellow.

Figure 10

The first past the post system is illustrated, by this two dimensional, horizontal section taken from Figure 1. Note that in such a section solid objects can touch, but cannot overlap. Suppose we wish to reduce this image to one dimension, as viewed from the front of the cube, the bottom of the figure above. The blue lines run up vertically from the bottom, stopping at the first colour that they hit. So, second from the left, blue is first past the post. With the two clear rectangles left and right being given the null colour – which some people prefer white, some prefer black – marking those segments where there is nothing above. 

Note now, that in a one dimensional view of a three dimensional world, rather than the two dimensional section we started with, objects can now overlap. Or at least, the brain using its prior knowledge and expectations, may assume some overlapping even if, strictly speaking, it cannot be seen. This is suggested in the lower reduction where we have used Powerpoint’s transparency feature. Which might be implemented either by adding or subtracting the colours in some way and using the result to colour the overlap. Or by randomly mixing the two colours (in some appropriate proportion) across the pixels making up the overlap, which amounts to much the same thing when there are enough pixels and the viewer is far enough away. And might, in any case, be how colour is implemented, as a mixture of dots from some base set of colours, perhaps four or five of them.

This one dimensional view can be converted to a two dimensional view up by sliding our horizontal section up from the bottom of the cube to the top. The series of one dimensional views can then be assembled and projected onto the appropriate side of the cube to give something like the first past the post view we started with at Figure 1.

And, we argue, a reasonable approximation to what gets onto the retina and from there into the brain when the owner of the retina is looking into that cube from that direction. A brain which often knows that the left hand side of the yellow object is in front of the right hand side of the blue object and that the right hand side of the yellow object is in front of the left hand side of the red object. And while we know quite a lot about how the brain gets that knowledge, we know rather less  about how it stores it – this being something of an open question, reference 4 and  LWS-R of refence 5 notwithstanding.

In which connection, we note that while gravity and occlusion are important in the two dimensional world of reference 4, neither has much traction in one dimension.

Figure 11

While this next figure is an error, with the line of colours at the bottom not corresponding to any vertical view from a line at the bottom of our cube, although, as it happens, it is roughly what one might get from a horizontal view taken from a line across the middle of the front.

But the line of colour does tell us that there are at least three objects in our cube, says what colour they are and gives us some clues, possibly misleading, about their relative sizes and positions.

And the brain, given that it gets input from two eyes with largely overlapping fields of vision, has information about their range, information which might be present in consciousness, even if it cannot conveniently be coded in the visual layer of LWS-R. So we do get the direction, altitude and range mentioned above. For binocular vision more generally see reference 7.

From the objects to the eye

Figure 12

Imagine for the moment that we have just one eye, marked by the small blue circle at the bottom of the eight panels of the figure above. The point being that, vagaries of light and texture aside, all eight scenes are all going to look much the same, with solid blue subtending the same angle at the eye. Much the same is true in three dimensions, with all kinds of different shaped solids subtending the same solid angle at the eye.

From the limited information available it is sometimes hard to know what shape something is or how far away it is – although bringing prior expectations into play, we usually have a shot at it and come up with a conscious percept.

Furthermore, there may be some difficulty about how many objects there are, illustrated in the second panel from the left in the bottom row. With one way around this difficulty being to move the head around slightly, with it becoming clear that part of the left hand object is moving around in front of part of the right hand object. Such movement might also give some more clues about the shape and nature of the object or objects concerned. While we have no chance at all with the first panel from the right in the top row; the near object neatly occludes the far object, and we probably don’t even know that this last is there. Fortunately for us, such neat occlusion is relatively rare in the real world.

Figure 13

More generally, where one object of one colour partially occludes another object of another colour, there is usually no difficulty knowing which object is on top - and out in the real world, it is unusual for two objects to have exactly the same colour and texture. So bottom right in the figure above, it is very likely that we have a green rectangle on top of a blue rectangle. And in the case when the blue rectangle has a tricky shape, perhaps negatively matching some feature of the green rectangle, for example bottom left, most of the time it is going to be clear which way around the two rectangles are; at least clear from the point of view of probability. Just occasionally there will be ambiguous pairings, perhaps involving a confused or camouflaged boundary between the two objects, perhaps as top left – where the blue line upper left and the green line lower right have been added for clarity. An ambiguity which is resolved in the top right hand pair by the distinctive pattern clearly belonging to the blue object, which is thus clearly on top. At least that is the most likely interpretation – which could be disturbed by some extravagant prior expectation.

An ambiguity which might well have been resolved top left by deciding that we actually have one object, perhaps composed of two parts, rather messily joined together.

Nevertheless, given the rarity of such oddities, we argue that where one object partially occludes another, we generally know that it is two objects and which is which. We know that object A is partially occluding object B, rather than the other way around. We get the organisation in space of our objects right. The rule of whole over part enunciated at reference 4 works nearly all of the time.

Figure 14

In the figure above, following Figure 2 above, or the third figure of reference 4, the brain’s starting position will be that the green area is the ground and that the white area is the sky. And given gravity, given that most objects need to be supported, the brain’s starting position will be that the blue objects are standing on the ground, with the black line possibly being added by the brain to show the line of contact.

And in the absence of other information, the brain will assume that the left hand red object is standing on its blue object, again with the black line added to show the line of contact. In the case of the right hand red object, there is prior knowledge about the red object which makes it likely that it is behind its blue object, perhaps flying through the sky. So no black line of contact. Instead, we have given the blue object some transparency, a trick that most brains can’t manage, although we have argued at reference 6 that it might have bash. One senses the large part of the ship which is underwater, even though one can’t actually see it.

There is a possibility that the red object is sitting in a hole cut in the top of the blue object, but in the absence of any evidence to that effect, the brain discards that possibility.

So the brain knows quite a lot about the scene in front of it. On the assumption that the storage of the visual part of the scene is segregated from that of the rest of it, the question is how much of that information is projected into consciousness in visual terms and how much as supplementary information? Does the brain, for example, add the black lines in the figure above to the image it projects into consciousness, in the way of a painter, or does it add some labels to some other layer of consciousness, in the way of LWS-R? Does it do nothing of the sort?

Figure 15

Given all of which, we propose that the eyes and the brain actually do something rather like Powerpoint. They arrange the objects of present interest against a background, in the figure above, pale green for the sky and darker green for the ground. They know that the green triangle is in front of the red circle is in front of the blue rectangle. That the grey oval and the dark brown arrow are behind the pale brown rectangle. That the yellow disc is up in the sky and the pale brown rectangle is sitting on top of the dark brown arrow. That the other objects are sitting on the ground.

But the relative positions of the left hand cluster and the right hand cluster are not so easy; there are fewer clues. The brain might be agnostic about this or it may have made a guess.

We have shown the pale brown rectangle as 50% transparent, not because this is how we would see it, but to remind us that what goes on behind the pale brown rectangle may be an open question, and even if we think we know the answer, we cannot actually see it. Does the dark brown arrow stop exactly on the boundary of the pale brown rectangle, or does it carry on behind it, as shown? Or is it actually part of the grey oval?

The real world

We now summarise relevant aspects of the real world and our place in it.

We live in a three dimensional world sat on the surface of the globe, which for present purposes can be thought of as being flat, at least locally flat. The ground below and the sky above. 

A world containing a variety of more or less persistent objects. Some of them can be thought of as being fixed; sat on the ground and not moving about. Fixed objects, fixed points which provide a frame of reference for the current frame of consciousness. While other objects move around, while remaining on the ground. Some fly around in the air. Some objects are alive, with plants being mostly fixed and animals being mostly mobile. 

Figure 16

Most of these objects are both rigid and opaque with well defined boundaries. This rigidity is not greatly disturbed by most of the larger animals being articulated, with limbs that move about: articulations apart, animals preserve their shapes. As do trees, despite their boughs swaying in the wind. And while there are objects which change their shapes, for example a growing puddle on a more or less flat bit of asphalt, a slick of oil on the surface of a pond, or a cloud against a blue sky, they are the exception rather than the rule. At least at the sort of scales that are relevant to us: we don’t usually have to bother about the goings on of protozoans, even the rather large one, appropriately called a chaos, snapped above.

We are around 2 metres high and most of our objects of interest lie in the 1 centimetre to 10 metres range, a range of three orders of magnitude. With the chaos above being of the order of millimetre.

For most purposes, the air can be absent; there is no need to take it into account. In any event, light can travel through the air, bounce off objects and arrive at our eyes to form two dimensional images on our retinas – with our eyes being far and away our most important sensory organ. The world we build is primarily visual. We are also primarily diurnal – even though our eyes come with good night vision capability.

We want to label the objects in view and we want to know where they are, this last being our present concern. Kind, shape, size, direction, and range.

All this visual thinking should be tempered by two further considerations. First, the white-tailed deer of reference 11. They do not have colour vision, are not much interested in things which do not move, but can spot very slight movements at surprisingly long range. A kind of vision which does not appear to have much in common with the foregoing. Second, the multi-modal work described at reference 12 and touched on at reference 13.

Conclusions

The eyes and the brain are good at reducing the three dimensional world to a two dimensional surface expressing colour and direction. They are good at object identification and the relative position of pairs of objects in the same part of the visual field. They have go at range. Using prior knowledge and expectations they can resolve a lot of ambiguity and fill a lot of gaps.

We suggest that all this is drawn together, from the three dimensional world, into a layered, two-plus dimensional world, along the lines of LWS-R. A two dimensional world which is not that far from that of Microsoft’s Powerpoint, with its simple geometric objects, very much along the lines of those of Figures 1 and 2, with boundary and fill properties and with ‘in front of’ and ‘behind’ relations.

And while the brain may be able to follow systems designers in using different techniques, different systems even, for different applications, we suggest that this two-plus dimensional world, based on what we can see, is what it does most of the time. The in-brain visualisation of the rotation of three dimensional crystals of references 14 and 15 is very much a minority sport – although it would be interesting to the talk to the lady concerned about her experience of same.

References

Reference 1: https://en.wikipedia.org/wiki/Map_projection. 

Reference 2: https://en.wikipedia.org/wiki/Rear_projection_effect. 

Reference 3: https://www.chineseshadowpuppetry.com/. 

Reference 4: http://psmv4.blogspot.com/2021/03/relationship-cues-in-painting.html. 

Reference 5: https://psmv4.blogspot.com/2020/09/an-updated-introduction-to-lws-r.html. Which include, inter alia, another outing for the theatrical metaphor.

Reference 6: https://psmv3.blogspot.com/2017/04/a-ship-of-line.html. 

Reference 7: https://en.wikipedia.org/wiki/Binocular_vision. 

Reference 8: http://psmv4.blogspot.com/2021/03/a-one-dimensional-structure-part-i.html. 

Reference 9: http://psmv4.blogspot.com/2021/03/a-one-dimensional-structure-part-ii.html.

Reference 10: How do the blind ‘see’? The role of spontaneous brain activity in self-generated perception - Avital Hahamy, Meytal Wilf, Boris Rosin, Marlene Behrmann and Rafael Malach – 2021. See Annex E.

Reference 11: http://psmv4.blogspot.com/2021/03/white-tails.html. 

Reference 12: The Merging of the Senses – Stein and Meredith – 1993.

Reference 13: http://psmv4.blogspot.com/2021/03/waking-moment.html. 

Reference 14: https://psmv3.blogspot.com/2018/08/the-myth-of-unconscious.html. Rotation towards the end.

Reference 15: https://psmv3.blogspot.com/2017/04/bragg-and-son.html. The talk in question.

Group search key: sre.