A Study on constructing POI（Points of interest） sequences in Cinematic virtual reality content

Ⅰ. Introduction

Over recent years, while virtual reality technology rapidly develops, research on storytelling in VR is being done orderly. As HMDs oriented to public market are propagandized, people do not satisfy with pure experience, researchers start to focus on designing narrative contents, i.e, how to build and tell story in virtual space better. Many studios with core purpose of researching storytelling in VR have been successively founded around the world, for example, the most famous 360 Google Spotlight Stories. With main research carrier of VR videos and animation, this studio has created many excellent works such as Pearl and Back to the Moon and discovered a series of new forms of storytelling in VR. Large-scale film festivals, for example, Venice International Film Festival, have also set with competition unit of VR story. Cinematic virtual reality (CVR) is to find VR with media loyalty in feature film.[1] The concept of CVR has been widely used in VR content production field. Different from VR game with core of interactive experience, CVR focuses more on storytelling, which means that telling a good story is the most important. Participants are immersed in a 360 sphere, their relationship with space is fixed and audiences cannot walk randomly in this 360 space.[2] According to Lescop L. Mateer, " CVR limits the level of control users have within the environment to choosing viewpoints rather than interacting with the world itself." In CVR narrative, user attention selection, gaze and information extraction are done through the design of POI sequences. Researchers on structure and writing skills of 360 storytelling in VR are still not enough. This paper, based on John Mateer described CVR, discussed the construction of POI sequences in CVR.

Ⅱ. Theoretical background

Ⅲ. Construction of POI sequences

3-1 POI classification under narratology

The classification for layer of structure of narrative works in narratology is greatly influenced by Russian formalism. Chatman made research on narrative works at two layers of story and discourse and tried to build a set of narrative structural modes.[12] to theory of structuralism, each narrative includes two parts, one is story, i.e, chain of contents or events, plus background and characters;second is discourse, i.e, expression, upon which contents are conveyed. This paper, based on Chatman's theory of story and discourse, has classified POI types in CVR into scene POI, events POI and discourse POI. Table 1 shows classification resulting from interaction between user's initiative and director's restriction.

Table1.

POI classification

1) Scene POI (Recessive cues)

Storytellingin CVR is a space narrative or scene narrative, and background description is significant in narrative. "Most scenarios may simultaneously provide multiple POI or cues, or even chaos. Recognizing POI is the first important." [10] Generally, when users enter 360 story space, the first step is scene perception and then they will understand story background and their roles. The scene perception requires visual and active exploration of users. Scenario POI is a recessive and implied narrative cues, so that director designs low visual restriction for users. Due to saccadic suppression in the course of saccade,[13] high-quality visual information can be acquired in the course of gaze only.[14] Film producers use visual saliency (generally defined to low-level elements of scenario, for example, contrast, hue and brightness, etc.), dynamic background element, Leading lines (perspective line, track line, visual line, etc.), user viewpoint and distance of object, object with semantic inconsistency and method of setting significant objects to build scene POI, attract users to discover actively and generate gaze.[15] (Itti, Laurent) For example, Sight line is an invisible visual leading line, but we are easily interested in others' sight. Like VR interactive cartoon, when the mouse in Buddy VR and hedgehog in Herry look at you, for empathy, you may pay attention to them. In addition, As unusual/weird scenes, objects with semantic inconsistency, for example, "fish floating in the sky of urban city", may easily attract attentions of audiences.[16] Visual saliency is also POI cue frequently used by CVR director. This is to attract attentions through bright colors and highlighted objects. However, when users see significant objects, relation between point of gaze and visual saliency will be weakened, i.e, significant and interesting cues will be more likely to attract attentions of users.[17] For example, objects appear repeatedly in film narrative or users see a cat staying in a table. Different users show individual difference for such significance. Therefore, film producer shall consider more about users' expectations and behaviors.

2) Events POI (Dominant cues)

Event POI refers to an active POI reminding users to pay attention to when events occur. [10] Event POI is a dominant and indicative story cue. In storytelling in VR, user's sight may not cover location of event, so that panoramic sound, which is a directional cue, is particularly effective to attract attentions. "Voice will prompt users to divert and locate roles and pay attention to important story events,"[18] for example, VR cartoon Lost. Visual guides, for example, spotlight and pop-up objects (firework, sudden magic), are also common event POI. Differently, these events shall occur in the range of gaze, otherwise, users may miss plots. To speak of, in multi-sensory storytelling in VR, sense of touch and smell can also be used as directional event POI to attract attentions of users. In addition, when designing POI weight, director shall also consider spatial distribution of kernel events and satellite events. According to understanding of French narratology expert Roland Barthes for kernel events and satellite events, kernel events are critical or turning points of story and cannot be omitted, while satellite events aim to supplement, enrich and complete kernel events which can be full and detailed. [19] Therefore, upon importance of plots, POI can also be further classified into kernel POI and satellite POI.

3) Discourse POI (Restricted cues)

Discourse POI is attention attracting cue independent of story and more inclined to a method or instrument. Different from traditional VR game, users in CVR cannot move in real time in 360 space, so that editing and camera stand motion are still important and significant. [20] We usually see in CVR, creator turns angles of camera by force to control user's range of gaze. For example, in Rain or shine, when the little girl went out from home, camera tracked and shot her from left to right, until she stopped in front of another scene POI (an ice cream truck).[21] Editing is usually made when narrative space changes in storytelling in VR. For example, in CVR Evely's Story and VR animation Pearl, producer realizes continuity of narrative space through editing.[22] Besides, editing and camera stand motion, CVR narrative has also generated unique modes, for example, view limitation, etc. Discourse POI is a restricted story cue, existing beyond the story and restricting range of user's attentions by force to narrative.

3-3 POI sequence

POI sequence is similar to picture sequence in comic and shots sequence in traditional film. Storytelling in VR, through building a cue chain of POI sequence, attracts users to distract their attention from a story cue to another and promote plot development. Lester C. Los proposes the Scene Perception & Event Comprehension Theory(SPECT).[24] He emphasized that the front-end of visual narratives involves two key processes that occur during each fixation: attentional selection and information extraction. Attentional selection determines what information to process during single fixations, and where the eyes will be sent for the next fixation. This is also the key link to maintain narrative coherence. According to this paper, building POI sequence shall consider contents of two dimensions, one is a combination of POI types, and the other is the layout of POI's position. Based on the (SPECT) theory, we constructed a POI sequence to guide the user's attention selection and information extraction model. See Figure 1.

Fig. 1.

Model of POI sequence to guide the user's attention selection and information extraction. The eye icon denotes the position of viewer gaze on the stimulus during a particular fixation.

3-3-1 Combination of POI types

1) Scene POI, event POI and discourse POI

Scene POI is recessive narrative cue; event POI tends to be a dominant cue; discourse POI is a restricted narrative cue. The degree of each POI in guiding users to distract their attention is different. The synergistic effect among the above three POI is indispensable for a film.

2) Kernel POI and satellite POI

POI，upon primary and secondary structure of events in the story can be subdivided into kernel POI and satellite POI. Traditional films distribute time layout of primary and secondary events through montage language. However, features of the VR space narrative allow primary events and secondary events to co-occur in the same space, and users can watch selectively. Kernel events develop throughout the whole story. [12]

3-3-2 Layout of POI’s position

In three-dimensional space, the freedom of gaze distribution enables audiences to look for story cues actively. These POI cues are arranged in 360° narrative space to attract users and distract their attention to different objects and then maintain story continuity. The director shall design spatial positions for these narrative cues in advance. Obviously, POI may be in the sight of users or beyond their sight. The layout of POI sequence’s position, upon a range of visibility and mutual positions, can be roughly classified into 1. Concentrated; 2. Scattered; 3. Coincident.

1) Concentrated

Fig. 2.

Concentrated POI spatial layout

2) Scattered

Fig. 3.

Scattered POI spatial layout

3) Coincident

Fig. 4.

Coincident POI spatial layout

Ⅳ. Research methods

In this research, issued CVR video contents are analyzed through qualitative content. Five VR 360 video clips are selected to discuss the construction of the POI sequence from a combination of POI types and layouts of POI’s position of POI sequence. The skills of storytelling in VR is in the process of exploration; narrative designs of these films are not mature. Therefore, we will limit research objects to POI sequences with high narrative continuity in CVR film plots. Table 2 has shown five contents selected for the case analysis.

Table2.

List of analysis for contents case

4-1 Design of POI sequence in The Invisible Man

The Invisible Man tells a story that two drug dealers were forced to play Russian Roulette with Frank(their debtor). The story took place in a narrow room, where the user played a role and was fixed on the seat. He or she can turn to the head to observe the surrounding environment. This film is a typical stage type VR story (like Back to the moon). The story was made in a small single room, and events POI sequences of figure performance was used to attract the user’s attention. The producer does not control through the discourse mode of limiting the range of visibility by force but has set scene boundaries based on scene resources. [25] A goods shelf was put behind the user. Because the goods shelf does not have any POI narrative cue, the user will only concentrate on the front table.(see Figure 6) Director declines scene POI and suggestively guides users to focus on the front.

Fig. 5.

Concentrated POI spatial layout

Table3.

POI sequence in The Invisible Man

Fig. 6.

Scattered POI spatial layout

4-2 Design of POI sequence in TheLoud House 360

The Loud House 360 is a 4-minute animation short film. It tells a story that children get up and prepare to go to school in the morning. Audiences go into the house and place themselves in a chaotic center. At the beginning of the film, the director has restricted POI quantity in scenes and set the unique scenario POI (a birdcage with a bird) that can attract the user’s attention at the end of the corridor. Moreover, then, the bird tweets and sister go out of the neighboring room and wakes all up. Children successively go out of their rooms, speak or perform. The process from birdcage to birds tweeting is the transition process from scene POI to events POI, which attracts audiences’ attention through interesting objects firstly and then controls their gaze through suggestive POI; as a result, to develop overall plots. The animation realizes narrative continuity through directional voices. As effective events POI, voices are scattered around the leading role and appear in sequence at a preset time by the director. (see Figure 6) When the current event ends, a new voice cue (speaking voice, door opening voice) reminds audiences to turn to voice direction.

Table4.

POI sequence in The Loud House 360

4-3 Design of POI sequence in ISLEOF DOGS: Behind The Scences

This short film is a propaganda video of 360 for the Isle of Dogs, face to face with the cast of dogs as they are interviewed on set, while the crew works around you to create the animation. In the background setting, the producer uses compositional lines (e.g., perspective line, intersection line) to enhance the central positions of actors. According to Figure 7, “dog actor accepting interview” (Point A) is kernel event POI and“working staff” (Point B, C &D) are satellite events POI, scattered on the other side of kernel event zone. Kernel events POI and satellite events POI are simultaneity, and users can watch selectively. The narration of the dog actor is a narrative cue so that users will not miss kernel events while watching satellite events.

Fig. 7.

POI arrangement in which kernel events and satellite events occur simultaneously

Table5.

POI sequence in Behind The Scences

4-4 Design of POI sequence in Special Delivery

“Emmy award-winning platform, GoogleSpotlight Stories, presents Special Delivery VR, an interactive experience byAcademy Award-winning studio.” This VR animation tells an adventure story that a guarder was disturbed by a mysterious stranger (Father Christmas) on the roof, and they chased each other. The sensor on the mobile phones enables users to interact with the story. When users divert their attention to various scenes, they can unlock potential events in the story. At the beginning of the film, the director has restricted users’ viewpoints and directions so that they will not miss titles. Later, the method of tracking shot is used to attract users’attention to events of “removing snow by guarder” and “finding Christmas ball,” while unlocking limitation for sight. The combination of discourse Poland events POI has maintained a smooth narrative.

The course of “chasing Father Christmas by guarder” in the story is kernel event POI of this film, which has attracted most gaze distributions of users. When users discover the surrounding environment, kernel events will stop, until users divert their attention back again. Director has uniformly designed several potential events POI in the space; for example, “a lady decorates Christmas tree,” “an old man plays the drum set,” “an old man feeds pigeons,” and “a lady plays the cello,” etc.[Figure8] Besides, the time range that can be triggered for subordinate events has been set; as a result, excessive satellite events will not be triggered to avoid interference on main events. Some cues are presented as background POI at the beginning. When trigger conditions are satisfied, hidden background cues will transform into dominant cues; for example, a little child gets up and bounces on the bed. The red PointA in Figure9 refers to the POI sequence of kernel events; the blue triangle mark refers to the POI of potential subordinate events.

Fig. 8.

Potential events POI in narrative space

Fig. 9.

Potential events POI

Table6.

POI sequence in Special Delivery

4-5 Design of POI sequence in Auto

Auto tells a story about an automatic car driver. Although this film is not mature in shooting skills, the combination mode of POI for the scene of “railway station” in the story is exceptionally smooth and suggestive.

Firstly, the director puts users on the railway platform, and then users start to discover scenes actively. Surrounding background resources, including train, road, lamps, and railway platform can be recognized. As compositional Lines, the perspective lined formed by road, lamps, and train enables us to pay more attention to vanishing points at both ends. As shown in Figure, as dynamic background cue, the “started train” enables users to pay more attention to the vanishing point (Point A) in the working direction of the train. [26] When the train disappears at the vanishing point, users’ gaze has not been ended. As a new POI, the film title “AUTO” (Point B) covers vanishing point so that our gaze time is prolonged. Later, the director applies editing skills to change narrative space from railway station to home of the leading role. The position of the leading role’s face (Point C) coincides with “AUTO” in the previous scene, and events POI of the couple’s dialogues are further developed. [Figure10], [Figure11]

Fig. 10.

POI sequence construction in Auto

Fig. 11.

Coincident POI spatial layout in Auto

Table. 7.

POI sequence in Auto

V. Conclusions

Through the case study, we found that because different POI had different influences on the user's attention selection, the sequence formed between them might have different narrative effects, thus affecting the user's behavior. This paper combines POI pairwise to explore the possibility.[Table8.]

Table. 8.

POI types combination and user behavior

These combinations, combined with the three spatial arrangements of POI, can generate more narrative possibilities.

When POI is concentrated arrangement, the sight of audiences shall be restricted to a certain range. The method of forcing user’s attention through limiting discourse POI, such as limited viewing angle/camera movement/fuzzy effect occlusion, will easily destroy user experience, for example, the prologue of Special Delivery. However, the method of reducing excessive scene POI and setting scenario obstacles in narrative space is more natural. For example, a wall, a piece of mist, or a dark zone is set at the back of users. Because of lacking POI, users will actively ignore sheltered zones. We indicate, concentrated layout mainly focuses on POI sequence, like the stage play, user attentions are attracted through figure shows and walking. Because actions are mostly shown behind audiences, the story will be easily understood.

Scattered POI layout shall consider POI types, the main primary and secondary structure of POI. When users are surrounded by scene POI, scene POI related to kernel events will be highlighted to better combine with subsequent events POI. Based on primary and secondary events, we can conclude three conditions: 1) only kernel events POI sequence exist; 2) kernel events POI and satellite events POI exist simultaneously; 3) potential satellite events POI may exist.

Except for the first condition, the latter two shall set kernel events POI sequence as watch zone and subordinate POI as secondary zone, and users can select to watch. Two POI sequences are usually distributed in different zones to avoid mutual interference. Like /Special Delivery/, satellite events POI sometimes is a potential event cue that is sudden and can be triggered. Different from the POI sequence chain of kernel events, satellite events are usually isolated micro-events.

Besides, in scattered POI layout, the sound is a special event POI. It has directional and narrative attributes and can reach a good synergistic effect with other narrative cues. When voice POI is a directional prompt, it can attract users to rapidly position voice source (for example, The Loud House 360) and combine with other events POI. When it is a narrative voice like dialogue/independent narration/prologue, users can understand story plot, while discovering scenes or watching other simultaneous satellite events, for example, ISLE OF DOGS. While improving narrative efficiency, the second condition can enhance user presence and, therefore, shall be paid more attention to by VR producer. We also indicate, distracting users’ attention may be an intention of the director, through multiple scattered POI layout, to create confusion or panic emotion.

Coincident POI layout in different spaces has a strong synergistic effect, but the precondition is to guide audiences for gaze for a certain time. Setting attractive scene POI, indicative events POI, tracking shot or limiting viewpoint, or other discourse POI can reach gaze effect. Later, editing discourse (fade-in fade-out / changing scenes, etc.) can be made to combine with a matching point in another narrative space. In this course, multiple POI synergistic solutions may appear, depending on narrative types and intentions of director.

Different POI sequences may have different effects. On the one hand, a producer can set multiply scattered scenario POI in space to create abundant scenes and then guide users to distract and pay their attention through directional events POI (e.g., sound, etc.) or discourse POI (e.g., changing camera position). On the one hand, setting prominent/interesting/significant scene POI enables users to discover and gaze actively and then matching front and rear POI’s positions through superposition can introduce subsequent events. According to this paper, the former is more applicable to the director’s intention of creating immersive scenes. It does not require to predict excessive behaviors of users, but the narrative rhythm is slow. The latter focuses more on internal relationship between narrative efficiency and narrative, but the director shall consider more about narrative logic/audiences’ behaviors/transition mode. However, how to combine POI depends on the director’s intentions and narrative types and shall be further studied.

This paper aims to do exploratory research on how POI sequences are constructed within cinematic virtual reality content. The synergy between the combination of POI types and layout of POI's position has formed different POI sequence. That is critical for maintaining story continuity and user presence. Different types of POI cooperate because of their different degrees of guidance to the user's attention choices. Combined with the POI location arrangements (concentration, dispersion, and coincidence), the construction of POI sequences has great potential in the narrative. We point out that the POI sequence is not a simple addition of narrative clues, but should take into account the director's intention, film type, causality, and user behavior. This paper provides theoretical tools and inspiration for the design of narrative cues in the production of CVR. The construction of the POI sequence remains to be further explored. How to design the rhythm and density of POI sequences? How to deal with the relationship between POI design and user expectations? It is the subject we will study next.

Acknowledgments

This paper has been conducted with the support of the “Design Engineering Postgraduate Schools” program, a R&D project initiated by the Ministry of Trade, Industry and Energy of the Republic of Korea. (N0001436)

References

• Mateer J. Directing for Cinematic Virtual Reality: how the traditional film director's craft applies to immersive environments and notions of presence[J]. Journal of Media Practice, 2017, 18(1): 14-25. [https://doi.org/10.1080/14682753.2017.1305838]
• Dooley K. Storytelling with virtual reality in 360-degrees: a new screen grammar[J]. Studies in Australasian Cinema, 2017, 11(3): 161-171. [https://doi.org/10.1080/17503175.2017.1387357]
• Li, Yi-meng, and Jong-woo Joo. "The change of animation narrative structure caused by virtual reality technology." Journal of Digital Contents Society(J.DCS),  20.3 (2019): 459-467. [https://doi.org/10.9728/dcs.2019.20.3.459]
• Riedl, M. O., & Bulitko, V. Interactive narrative: An intelligent systems approach. AI Magazine, 34(1), 67-77，2012. [https://doi.org/10.1609/aimag.v34i1.2449]
• Cruz-Neira C, Sandin D J, DeFanti T A, et al. The CAVE: audio visual experience automatic virtual environment[J]. Communications of the ACM, 35(6): 64-73.1992, [https://doi.org/10.1145/129888.129892]
• Slater M, Usoh M, Steed A. Depth of presence in virtual environments[J]. Presence: Teleoperators &Virtual Environments, 3(2): 130-144.1994, [https://doi.org/10.1162/pres.1994.3.2.130]
• Heeter C. Being there: The subjective experience of presence[J]. Presence: Teleoperators &Virtual Environments, 1(2): 262-271.1992. [https://doi.org/10.1162/pres.1992.1.2.262]
• Itti, Laurent, Christof Koch, and Ernst Niebur. "A model of saliency-based visual attention for rapid scene analysis." IEEE Transactions on Pattern Analysis & Machine Intelligence, 11: 1254-1259. 1998. [https://doi.org/10.1109/34.730558]
• Syrett, Hannah, Licia Calvi, and Marnix van Gisbergen. 2017. "The Oculus Rift Film Experience: A Caseon Understanding Films in a Head Mounted Display." Intelligent Technologies forEntertainment: 8th International Conference, INTETAIN 2016, Utrecht, TheJune 28-30, 2016, Revised Selected Papers, 197-208. Springer International. [https://doi.org/10.1007/978-3-319-49616-0_19]
• Jessica Brillhart, “In the Blink of a Mind — Attention“, https://medium.com/the-language-of-vr/in-the-blink-of-a-mind-attention-1fdff60fa045
• Nielsen, Lasse T., et al. "Missing the point: an exploration of how to guide users' attention during cinematic virtual reality." Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology. ACM, 2016. [https://doi.org/10.1145/2993369.2993405]
• Chatman, Seymour Benjamin. Story and discourse: Narrative structure in fiction and film. Cornell University Press, 1980.
• Matin, E. (1974) Saccadic suppression: a review and an analysis. Psychol. Bull. 81, 899-917 [https://doi.org/10.1037/h0037368]
• Volkmann, F.C. (1986) Human visual suppression. Vision Res. 26, 1401-1416 [https://doi.org/10.1016/0042-6989(86)90164-1]
• Matin, E. (1974) Saccadic suppression: a review and an analysis. Psychol. Bull. 81, 899-917세 [https://doi.org/10.1037/h0037368]
• Rayner K, Castelhano M S, Yang J. Eye movements when looking at unusual/weird scenes: Are there cultural differences?[J]. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35(1): 254.2009. [https://doi.org/10.1037/a0013508]
• Oliva, A. et al. Top-down control of visual attention in object detection. IEEE Proceedings of the International Conference on Image Processing (Vol. I), IEEE, pp. 253-256,2003.
• Jerald, Jason. The VR book: Human-centered design for virtual reality. Morgan & Claypool, 2015. [https://doi.org/10.1145/2792790]
• Wallace, and Martin. Contemporary narratology. Peking University press, 1990.
• Lescop, Laurent. "Narrative grammar in 360." 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct). IEEE, 2017. [https://doi.org/10.1109/ISMAR-Adjunct.2017.86]
• VR 360 video contents of ‘Rain or Shine’, Available:, Available:https://www.youtube.com/watch?v=QXF7uGfopnY&t=127s
• VR 360 video contents of 'Pearl', Available: https://www.youtube.com/watch?v=WqCH4DNQBUA&t=245s&pbjreload=10
• Cohn, Neil. The Visual Language of Comics: Introduction to the Structure and Cognition of Sequential Images. A&C Black, 2013.
• Loschky L C, Larson A M, Smith T J, et al. The Scene Perception &Event Comprehension Theory (SPECT) Applied to Visual Narratives[J]. Topics in cognitive science, 2019. [https://doi.org/10.1111/tops.12455]
• Lynch K., The Image of the city, Cambridge, Mass, MIT Press, 1960
• Borji A, Feng M, Lu H. Vanishing point attracts gaze in free-viewing and visual search tasks[J]. Journal of vision, 16(14): 18-18, 2016. [https://doi.org/10.1167/16.14.18]