Another look at frames

Figure 1

Contents

• Introduction
• A recap on layers
• Flickering
• Films, the stream of consciousness and the passing of time
• Films: the block model
• LWS-N: the block models
• Frame: the weak analogy with films
• Frame: the analogy with 1970’s computing
• Transitions
• Conclusions
• References

Introduction

This post was prompted by thinking about the frames of consciousness (of LWS-N of reference 6) which would be involved in one of the counting tasks that I post about from time to time; in particular counting the holes along the length of a stretch of overhead shelf in a train, as in the snap above, the subject of a post to come. And some preliminaries are to be found at a couple of posts just past, references 4 and 7.

An earlier look at frames is to be found at reference 1. Where a sequence of frames amounts to a take and a sequence of takes amounts to a scene. Terms lifted from the world of film, but used in a rather different way. For example, in a making a film one might take the same bit of a scene, the same shot many times, until one gets it right – while the brain only gets one go, at least as far as consciousness goes. In what follows we shall be taking a look at what films do, and how that differs from what brains do.

A recap on layers

In LWS-N, consciousness is hypothesised to arise from the activation of one or more of its layers, with just one layer being enough. The layers might be thought of as being both additive and differentiated; or put another way, they are experienced both jointly and separately, a seeming contradiction which seems to be at the heart of the consciousness puzzle. It crops up, for example, in reference 8, recently mentioned in reference 7.

Layers are two dimensional objects, built on an approximately two dimensional sheet of neurons. And at any one time, any one layer is either space-like or time-like. That is to say it is either mapping something – like what we see – in two dimensions, a snapshot in time, or it is using one of the two dimensions available for time, usually for sound – particularly the peculiarly human sounds of speech and music – which is bound up with time in a way that sight is not, the fact that the world we see moves around in time notwithstanding. The same is true of touch, a lot of which can only exist in time. And to complicate things, some wine buffs talk of taste having a time element too, or at the very least a foreground taste and a background taste.

Again at any one time, layers mostly hold data of just one modality, with important modalities being our five senses: sight, sound, taste, smell and touch. All of which, apart from touch, both starting and finishing with the head.

Each layer is organised into a number of foreground objects and a background object. Exceptionally, there may just be the background object. For present purposes we leave aside the decomposition of layer objects into parts and the linking of layer objects from different layers into composite objects.

Presence of data in a layer is a necessary but not sufficient condition for it to be conscious. But presence of an activated object in a layer is a necessary and sufficient condition for it to be conscious. Usually the level of activation will vary from object to object and from object to background. And frequently there will be one stand-out object, the object being attended to. The object of consciousness; what it is that one is conscious of. But we do allow other possibilities.

Roughly in the way of the two sorts of movement of the eyes – saccade and smooth pursuit – layers can change in one of two ways: creation or maintenance, otherwise construction or maintenance. From where we jump to the better analogy of the transition from construction to maintenance of something like a building or a computer system, where we find a certain amount of untidiness at the boundary. Bottom-up untidiness which is managed top-down by having a formal sign-off which takes one from one state to the next.

When an old layer is replaced by a new-build layer, there will be a gap in time between the old layer and the new. If for no other reason than writing the new data to its new location takes time. To which must usually be added the time needed to activate the new layer so that it becomes conscious. A gap which might total 100’s of milliseconds.

Layers need to do work to stay alive. Or more precisely, they either need to be getting input or to be being activated. Otherwise they decay with a half life of the order of tens of milliseconds. So when we wake up from a period of unconsciousness – sleep, coma, anaesthetic, whatever – we have to start from scratch. With our impression being that the layers come online one after the other, over a period of time; not usually like switching on a light. A pre-existing toothache, for example, might only come online some seconds after one first reaches consciousness.

But we do allow the persistence of layers through periods of unconsciousness. Indeed, stuff might make it to a layer, but never make it to consciousness.

So layers have a life history: they are created, they exist for a while and then they fade away. With perhaps half a dozen of them on the go at any one time, in any one healthy, conscious brain.

Layers have, to a considerable extent, an independent life. A layer can go through its life without much regard to what is happening in other layers. That said, there are at least two factors which tend to pull layers into line.

First, there are environment variables. Layers can only exist at all if the relevant environment variables are right, perhaps within quite narrow limits, and so all layers will rise and fall with those environment variables.

Second, there are links between layers. The information in one layer is supplemented by information in another. And this supplementing requires direct links which say which bit of one layer is related to which bit of some other layer. And requires something to coordinate activation. It is no good knowing that this blob is a member of the Meixnerididae family unless these two bits of knowledge are brought together by some kind of joint activation.

Figure 2

Note lastly that an old fashioned film shot on and developed onto film stock had just two layers, clearly visible to the eye, a wide band for sight (above in the figure above) and a rather narrower band for sound (the pink below) – a band which appears to proceed without regard to the frames above. Whereas in LWS-N the number of layers varies according to circumstance, although sight and sound do remain separated, on their own layers.

Flickering

Figure 3

Vision tends to dart about the world outside, to dart about those parts of the body which can be seen, and this darting about can be tracked using eye tracking devices, as in the figure above. Leaving aside movement in the world outside, the result of this can be simplified down to a sequence of stills, with this being part of the motivation for frames of consciousness.

One’s first thought is that the other senses can’t do this. You hear what you hear, you taste what you taste – and that is the end of it. Although one rather fanciful complication arises from the wine buffs mentioned above. If we suppose tastes to be arranged linearly – in the way of the wavelengths of light or sound – and the taste buds to be arranged in a corresponding line from front to back of the tongue – there would be a temporal element to taste as the gulp of wine sloshed down over the tongue on its way from the mouth to the throat.

A less fanciful thought is that while the more or less external retinas can select input by pointing, the internal brain can do selection after input. So the brain can choose which fragments of the great slew of stuff coming in that it attends to. And this might be as jumpy as the eyes. A twinge from the toes, an itch behind the ear, a few bars of music from the piano upstairs, a spot of inner thought about snowdrops and so on. If one lies in bed of a morning, in the dark, paying attention to oneself, one is sometimes surprised at the amount of jumping going on. A jumpiness which is usually suppressed later in the day by concentration on activities which requires attention. Or which at least require the brain not to be busy on something else.

From which we offer the suggestion that a frame of consciousness is a period of time in which one is attending to something, external, surface or internal, typically for a second or so. Quite often, because we are vertebrates and vertebrates are visual animals, but by no means necessarily, triggered by a saccade. And atypically, some people are able to attend to more than one thing at once, perhaps two things, physically close together. And in the case of the counting task we started with, maybe the trick is to be able to slide one’s attention along, in this case, the line of holes, without attention wandering off anywhere else: in other words, an unusually long frame.

The layers holding the conscious information about the new object of attention will need to be recompiled – with this recompilation being the motivation for the compilation box in Figure 8 below. This is true irrespective of which layers held the old object of attention. And it seems likely, in so far as they need, as it were, to be turned off, that any old layers which are not also new layers, will also need to be recompiled. They might, alternatively, just be recycled, that is to say deleted.

Films, the stream of consciousness and the passing of time

While it is true that a good, well-edited film largely drives the stream of consciousness of the viewer, indeed of each viewer, not entirely, but largely separately and independently, it is also true that things can be done in films, done with films that the brain cannot usually manage in real life. A film can jump from place to place. It can jump from one point of view to another. Perhaps points of view which one would be hard put to manage in the ordinary way of things. It can zoom in on things and zoom out from them. A film is a manufactured thing, assembled from all kinds of bits and bobs from the cutting room floor. While the brain’s stream of consciousness is more like what one would get from a body-cam, perhaps the 1270p Super HD Professional Body Camera with IR Night Vision from Zetronix (of reference 5).

Although, all that said, as is explained at reference 2, classical film editing, in many ways, apes what the brain does as the eyes flit about the changing scene in front of them – and this is why it works.

Generally speaking, during a scene, a film runs at roughly real time. The actors and actresses do in five minutes what might reasonably be done in five minutes. But there can be long intervals of real time left out between scenes, which means that the film can span much more time, perhaps months or years, than its – say – 100 minutes running time. A film is not what happens at one time in one place (as favoured by cognoscenti of classical French theatre) – and I dare say there are arty directors who try to make films which portray something in real time, with running time being real time. As would indeed be more or less the case when one films match of the day.

Figure 4

In this graphic we have tried to suggest how a film might play with time. On the horizontal axis we have real time, the running time for the film. On the vertical axis we have time in the film. So, for example, we have it that 30 minutes into film we are 50 minutes into the 300 minutes of life that the film is portraying. 70 minutes into the film we are 250 minutes into the portrayed life. Most of the time films run at roughly the same speed as life, but they can both slow down and, more often, speed up – this last by missing lots of time out, perhaps when the leading character might reasonably be thought to be sleeping – suggested here by the near vertical sections of the plot.

Figure 5

The film can also do flash backs – or indeed flash forwards, this being much rarer. This is suggested in the sketch above, with the film going back to the beginning for a while when it is about 60 minutes in. There is clearly plenty of scope for playing elaborate games with time.

Figure 6

While the subjective experience is not that much like what the bodycam mentioned above does either. The experience does not flow evenly and steadily along, at least not most of the time. It comes in fits and starts, with a lot more being packed into some minutes than others. I associate to the heightened awareness that some professional rock climbers talk of. There might even be short absences, with petit mal being a pathological example. With the break of visual continuity when one blinks being rather different.

Under our hypothesis of frames, the frame count per minute might be a proxy for the amount going on, with the sketch above suggesting the sort of variation we imagine is usual, again taken over a running time of 100 minutes.

Film: the block model

Figure 7

This post is not directed at film buffs and what follows may be a bit wide of the mark. But it hopefully serves to motivate our analogy. And to the extent that it is about films, it about the sort of films which were made on film stock, on reels of film. From which things have now largely moved on to the world of digital with everything.

During production the film is broken down into scenes, with scenes taking place at one time, on one set or in one location. The scene is broken down into shots, with shots involving the continuous running of one or more cameras and microphones. Typically, there are several takes of any one shot; sometimes just one and less often many, sometimes even a hundred or more. Each take results in a continuous sequence of frames on the film stock. A lot of takes are thrown away, but some are developed and make it to the cutting room.

During editing there, clips are snipped out of takes and assembled into film scenes, with these last being only loosely associated with production scenes. Most takes are not represented in clips at all and very few takes are used in their entirety.

And running alongside the frames of vision, we have the ribbon of sound: a ribbon which is not organised into frames at all: indeed, as is explained at reference 2, continuous sound running across the clips often helps to stick them together from the point of view of the film-goer.

Note that while the brain does not do takes, it does do choice of the cutting room floor variety. The brain can select for projection into consciousness from among the various material coming in. Indeed, only a proportion of this last can make it to consciousness, and only a proportion of that actually does. The conscious brain does not have much to do, for example, with the muscular movements of the stomach. But that still leaves plenty of choice.

LWS-N: the block models

Figure 8

Figure 9

Figure 10

The brain knows about objects – for example, houses, lamp posts, people and pets – some in a generic way, some in a particular way. That is to say, for example, a particular person rather than people in general; a particular carpenter rather than carpenters in general. Roughly speaking and leaving aside what might happen in the brain at large, in memory, objects are expressed in LWS-N as layer objects. Several layer objects, linked together from several layers, will usually be used to describe a single real world object. Such a group of objects is called a composite object.

An important element of LWS-N consciousness is the object of attention, typically but by no means necessarily a composite object, with the electrical field resulting from the activation of the relevant layers amounting to subjectivity, to consciousness. Several layers may, will usually, co-exist in consciousness. Such layers may or may not be related, connected by composite objects.

The object of attention is connected to a frame of consciousness by a compilation. The compiler being shown in green in Figure 8 because the compiler is a chunk of processing capability, rather than something of which there are many instances.

The usually short period of time, often of the order of a second or so, for which an object is attended to, is a frame. Given the work that has to be done, there are gaps between frames: the switch from one frame to the next is not, cannot be, instantaneous. By definition, there is no subjective experience during the gaps and we suppose that one is not usually conscious of them, although clues about the passage of time will start to creep in to the frames following when the gaps get long or when there is a waking up period, a transition from unconscious to conscious, after a long gap.

The same object may be the subject of successive frames, perhaps from a slightly different point of view, perhaps to give the eyes a rest. Alternatively, the transition between the two may be a frame in its own right. And any one memory object may figure in consciousness many times during its working life.

The longer saccades of the eyes will take place during these gaps between frames, with the retinas not sending useful signals during such a saccade. We see what there is before a saccade and what there is after, but nothing in-between. And in the case that there was a visual object of attention before, there will always be a new object of attention after. Noting that the object of attention may not be visual, with visual processing taking place in the background. One might be walking along the street but one is thinking about, looking forward to, attending to the next black pudding.

Frames are organised into takes and takes are organised into scenes, while a layer may be bound to a frame, a take or a scene. A new frame nearly always involves a new object of attention. A new take always involves at least one new layer. While a new scene is a new beginning, with all the constituent layers being created from scratch.

Frame: the weak analogy with films

Figure 11

The term frame was taken from film, the sort of moving film made up of a long sequence of frames, each one amounting to a still snapshot – either in negative form as taken or positive form for projection. This is suggested in the figure above. Here, the division of the visual world into frames is arbitrary, at least driven by the mechanics of the camera rather than anything out in that world and each frame represents a slice of time of constant length, in the past often one 24th of a second. That said, one might tune the mechanics of the camera – perhaps the shape or speed of the rotating shutter – to take account of whatever it was in the world that one was filming. A tuning which might vary from film to film, but certainly not from frame to frame.

Figure 12

When projecting a film, the total duration of the gaps may be as long as that of the frames. Indeed, a film stock based projector is only actually projecting light about half the time, reducing the amount of light reaching the screen to that extent – with the balance presumably ending up as heat, given that the light bulb (or whatever) is on the whole time. Part of this is about the need not to project light through the film stock while it is actually moving, while it is moving from one frame to the next, which would be confusing. This is the business of another rotating shutter, perhaps something like that snapped above, with the rotation of the shutter cunningly linked to the movement of the film stock across gap between the light source and the lens.

Note that the images of objects moving relative to the camera will be blurred by their movement during each exposure – during which, for example, a car on a road is likely to have moved some centimetres. On which see reference 7.

In LWS-N, a frame is neither fixed nor arbitrary in this way, and both start and end points are context sensitive, with duration of frame varying from frame to frame. The frame captures for consciousness what is happening while the brain is attending to something, attention which might be broken if too much is happening, if, for example, there is too much movement, too much change, too much unpredictable change. It might also be broken if too little is happening, as the brain is not keen on statis. The drivers here include the something being attended to and the world around it, the constraints of the LWS-N compiler and the activation machinery which goes with it. The present thought is that frames will usually be of the order of few seconds in duration, say less than five, that is to say maybe two orders of magnitude longer than the frame of a film.

Compilation: the analogy with 1970’s computing

When Turing invented his famous machine in the middle of the last century, one wrote programs for it by punching holes in paper tapes. By the time that I came along in the 1970’s, the world had moved onto layers. One wrote one’s program in something called Fortran, not that far removed from natural language. The task might be, for example, reading a magnetic tape containing hundreds of thousands of death records and analysing them by age at some specific date (not always the same as age at death). This source code would then go through a short sequence of translators, or compilers, taking one through semi-compiled code, compiled code, consolidated code and lastly machine code. Adding in various standard chunks of logic from stored code along the way, perhaps, for example, the logic needed to turn a number into a logarithm, denoted in Fortran by the short logic-free string ‘log’. This machine code was loaded into a segment, perhaps some tens of thousands of bytes long, in computer memory. From time to time the computer would execute that code – during which time, generally speaking, the action of the code was constrained to fall within its own segment. No interference with other segments, which were, in all probability, doing something quite different, quite unrelated.

In round terms, one could not change the machine code, one just had to let it run, to let it do its thing. The person writing the source code had little if any knowledge of the machine code generated from that source code. And even if one knew how to fiddle with the machine code, it was almost certainly faster and less error prone, to start over. So if one wanted to make a change, or correct a mistake, one went back to the source code, made the necessary changes and went back through the compilation process. Slow, but sure.

So roughly speaking, one had a whole lot of intelligence going into the production of compiled code. That code then ran for a bit. Then one tried again.

The present hypothesis that consciousness works in the same sort of way. All kinds of input is collected up, turned into a frame of consciousness and fired up, activated to generate the subjective experience.

In the meantime all kinds of new input is coming in. To the extent that this new input fits with what has gone before, it might be used to amend the current frame. If new is very new, the brain opts to build a new layer or layers, possibly a new frame, sometimes experienced as a jump – small or big – from the old to the new frame. But more often, one is quite unaware of these jumps. When one’s attention has moved to the meat, one quite forgets, one is quite unaware that one had been attending to the potatoes. One is conscious of what one is conscious of, not what one was conscious of.

Transitions

It is possible, provided one does not force it, to divide activities into substantive activities, like changing the tyre on a car, and transitional activities, like going back to the house to make a cup of tea. Or driving down to Devon from London for a holiday. In a film, these transitional activities might be important, might be the subject of their own scenes. Equally, they might be just suggested, maybe skipped over, with the viewer’s brain left to work it out for itself. And much the same is true of real life, with some transitions being substantial activities in their own right and with others barely making it to consciousness at all.

Suppose we have a scene in a country house in Derbyshire, followed by a scene involving some of the same people in a town house in London. These two scenes might be quite self contained in themselves, perhaps involving no more than the one room each.

But for one of the people, moving from Derbyshire to London, there is the business of getting from one to the other. Perhaps sitting in some rackety, smelly, horse drawn carriage for hours, if not days. Or if we shrink things down a bit, that of getting upstairs from the dining room to one of the bedrooms. This last process might be almost unconscious, a rapid transition from one substantive activity to another. But the first process will not be unconscious; it is a substantive activity in its own right. While a film, for reasons of its own, might make more of the second than the first.

In a film, a transition may be no more than suggested, or perhaps skipped over altogether. Or, alternatively, the camera may carefully follow someone from one room to another, not breaking the continuity of a take at all. An effect one might get subjectively in, for example, that one is reading a book (or perhaps, more realistically these days, listening to the radio) and one carries on reading the book as one moves from one room to another. The primary subject of attention, the book, is maintained throughout. The layers of LWS-N supporting the reading of the book are maintained throughout. Remembering here that there is some leakage in that one might mostly be looking at the words on the page, but one’s eyes do pay some attention to the surroundings, if not one’s full attention, except in the case that something goes wrong, there is some failure of prediction or there is an interrupt.

One might characterise the difference between frames, takes and scenes by the amount of new input, the amount of change that can be accommodated. On the other hand, all subjective experience is expressed as frames, takes are only sequences of frames and scenes are only sequences of takes, so we cannot place any limit on the amount of change in a frame, or on the amount of change from one frame to the next. What we can say is that a frame which includes a lot of change, on the ground as it were, is apt to be blurred, is apt to leave out a lot of detail. Or that a take which includes a lot of change might include the start and end points and rather gloss over what lies between. The brain just does the best it can in the time available.

It is clear that handling movement is both important – good processing of moving objects was clearly important back in the good old days when we were hunter gatherers – and difficult. With part of the difficulty being the need to do more than move the pixels about in whatever structure supports their projection into consciousness: the pixels need to move together with their supporting scaffolding of supporting information. A crude analogy might be that the brain has to be able to move the diagram which is has superimposed on the image with the image – with the shape nets of LWS-N having something of the diagram about them.

A classification of the visual scene, in ascending order of difficulty, is offering in the figure following.

Figure 13

Difficulties which films have managed to overcome with great success. So there is perhaps something to learn from them.

Conclusions

Where we seem to have got to is that a frame in LWS-N is a lot bigger than a frame in a movie, in fact more like what we have called a shot. While a take in LWS-N is somewhere between the shot and scene of a movie.

Figure 14

But so far, the distinctions between frames, takes and scenes are rather soft distinctions. Pushed down onto what the brain is doing and much as pushing up from the bottom. This will probably changes as our notions firm up.

References

Reference 1: http://psmv3.blogspot.com/2017/06/on-scenes.html.

Reference 2: The Illusion of Continuity: Active Perception and the Classical Editing System – Todd Berliner and  Dale Cohen – 2011.

Reference 3: Yarbus, eye movements, and vision - Benjamin W Tatler, Nicholas J Wade, Hoi Kwan, John M Findlay, Boris M Velichkovsky – 2010. My source for Figure 3, originally from a paper by Alfred Yarbus, a Soviet Russian active in the 1950’s and 1960’s.

Reference 4: http://psmv4.blogspot.com/2020/01/moving-blocks.html.

Reference 5: https://www.zetronix.com/blue-line-police-body-camera.html.

Reference 6: http://psmv3.blogspot.com/2018/01/an-introduction-to-lws-n.html. A slightly out of date introduction to LWS-N. Searching for ‘sre’ will produce some more recent material.

Reference 7: https://psmv4.blogspot.com/2020/01/moving-cars.html.

Reference 8: Consciousness: here, there and everywhere - Tononi & Koch – 2015.

Group search key: sre.

Note

Figure 8 was corrected on 5th February, 2020.