Design and Implementation of Interworking System for Device Interaction in a Virtual Reality Setting

2-1 Virtual Reality

Virtual Reality(VR) refers to a specific environment, situation, or technology that is created by artificial technology using a computer that is similar to the real world but is not real. The created virtual environment or situation stimulates the user's five senses and allows them to experience the same spatial and temporal experience as the real world, freeing the boundaries between reality and imagination. In addition to being simply immersed in virtual reality, users can interact with things implemented in virtual reality, such as adding operations and commands using real devices. Generally, after wearing the head-up display (HMD), the external view is blocked and the feeling of immersion is provided through the screen in the HMD. It can be seen that existing 3D / 4D contents are provided as a new space. However, since the content is mostly personal, it is limited in the variety of contents. It is also pointed out that dizziness can be caused by adjusting the visual sense to generate immersion. Recently, HMD devices have been offering a variety of products ranging from HMDs for mobile phones that can be experienced at low prices to high-quality, light-weight, and wireless functions but at relatively high prices [2].

2-2 Augmented Reality

Augmented Reality(AR) is a field of Virtual Reality(VR), which is a computer graphics technology that synthesizes virtual objects or information in a real environment to make them appear as objects in real environments. As a result, augmented reality is a technique of overlapping virtual objects on the real world viewed by the user. Augmented Reality uses a virtual environment created by computer graphics, but the main area is the real environment. Computer graphics only serve to provide additional information for the real world. By overlapping the 3D virtual image with the real image that the user is viewing, the division between the real environment and the virtual screen is make it ambiguous[3].

VR technology allows users to be immersed so that they cannot see the actual environment. However, AR technology, which is a mixture of real environment and virtual objects, allows the user to see the real environment and provides better realism and additional information. For example, when you look around with a smartphone camera, information such as the location of a nearby shop and phone number is displayed as a stereoscopic image. Compared to a VR in which information on an environment that does not exist in reality is displayed through a display device, AR is a form in which virtual information is added to the environment that the user is currently viewing. So they are interested in advertising, management, maintenance, and medical care.

2-3 Mixed Reality

Mixed reality technology is a technology to show virtual world having additional information in real time in the real world as one image. The difference from augmented reality can be whether it is interaction or not. Augmented Reality simply plays the role of displaying virtual information on the real world. Mixed reality makes it possible to interact with the information seen, to manipulate the objects and to enable real-time interaction between the reality and the virtual environment. Research on mixed reality technology that combines the real environment and the virtual environment has been underway full-scale research and development since the latter half of the 1990s. There are not many instruments that support mixed reality at present, and it forms an expensive price. Although it remains at the research level, it will replace the virtual reality technology within a few years[4].

2-4 Modeling and Simulation

The academic definition of Modeling and Simulation(M&S) has not yet been well established. Given the importance of M&S's industrial applicability in recent years as ICT contributions increase, M&S concepts, terminology, and There is an increasing need for a formal academic definition of the range.

The Department of Defense(DoD) defines M&S as "An activity that creates a model that expresses systems or entities of interest, phenomena, or processes through physical, mathematical, or logical methods and analyzes them statically and dynamically." In this context, domestic research defined M&S as a time sequential representation of the system of interest and the operation principle of the system. The National Science Foundation(NSF) highlighted M&S activities as a means to streamline national R&D and defined M&S from an engineering perspective. According to the NSF, M&S builds the engineering system and predicts physical events and responses using computational methods.

As is evident from the definition of M&S, M&S is primarily a technology operating in a digital environment, is closely related to the development of various computer science fields, has a multidisciplinary nature that is embedded in various scientific fields such as arithmetic. The term "arithmetic" M&S is not only a field of engineering and science, but also of various kinds of research such as science, social science, business, medicine, education, etc. It has the characteristic that it can be applied to several fields where the field can come in and experiment with the dynamic model.

When these engineering and M&S technologies were introduced into the military and private enterprise, we would substitute the physical model of the real world to create a simulated product in digital space and predict its performance. From this point of view, engineering and M&S can be defined as "the virtualization of the design stage of a product, which is an important step in product development, A technology that can significantly reduce the cost and time of product development by replacing physical product manufacturing and experimental activities with virtual product modeling and analysis functions"[5][6].

2-5 Virtual Training System

A VR training system is a system that provides an experience that is difficult to experience in the real world by wearing an HMD and using VR technology. For example, training such as military, terrorist attacks, and fire suppression. This training requires a process to solve various problems in order to perform. These problems are as follows. First, it takes a lot of money to do the actual training once. Second, training should be conducted at certain times. Third, it is necessary to secure space for training. Finally, there is always a safety risk. Therefore, it takes a lot of cost, time, and place to carry out the training, and unexpected safety problems occur during the training, and it is difficult to carry out a lot of training because it can cause the human injury. As the number of training decreases, the training skill decreases, which leads to a vicious circle that can not achieve high performance in a practical environment.

A training system using VR can solve all of these problems. Obviously, the initial cost and space for building a VR training system is needed, but it can be constructed at a lower level than the actual training. Futhermore, a once-constructed system can be freely trained, with no extra cost, no time limitation, less space, relatively high safety, except for maintenance costs. This leads to frequent training times and ensures high proficiency. In the early days, simulators using displays were popular, but this was only for one-way education. Recently, a variety of training systems have been introduced in combination with VR technology that provides an immersive experience through interaction. It has a higher educational effect compared to the previous simulator, but still has a limit of single person training system. Since VR devices using HMD have limitations in view, it is difficult for many trainees to receive training in one space. In order to solve this problem, motion capture technology using motion camera is introduced, but it is too expensive to produce the system and it is difficult to supply the system. In this paper, we propose an mutually interactive middleware system for multiple training system through real-time data interworking between HMD devices using sensors, as shown in Fig. 1.[7][8].

Fig. 1.

Virtual training system for two more persons

3-1 Middleware

Middleware is collectively known as software that bridges the gap between various hardware, network protocols, application programs, local network environment, PC and operating systems. This paper defines middleware as a software system that coordinates the actual location information of each device between different VR devices [9].

3-2 System Configuration

This paper utilizes two units of HMD designed by HTC VIVE. Their specifications are shown in Table 1 below.

Table 1.

HMD Specifications

The following is the specifications of the middleware system and system for running VIVE (HMD).

Table 2.

System Specifications

4-1 Overall System Configuration Diagram

The overall system configuration is shown in Fig 2 below.

Fig. 2.

Overall System Configuration

It consists of a separate system for driving the HMD and a middleware system that relays each driving system and process and transmit the information provided by the HMD. HTC Considering VIVE officially supports SteamVR, the virtual training system is designed using Unity and the middleware system is composed of C++. Finally, the SteamVR is used to link the HMD devices.

4-2 Keypoints for Interaction

In unity, the HMD object information is generated in the form of a three-dimensional Vector. Basically, it processes and uses the data including three-dimensional position (POS), the left and right angles from the objects (DIR) and the upper and lower angles (ROT).

The procedure is as follows.

- HMD object data is transferred to the middleware using the TCP socket communication.- The middleware system processes the data and sends it to the connected HMD operating system.- Each HMD operating system applies the data received from the middleware to each object.

The main source code is shown in Fig 3 below.

Fig. 3.

Main Packet Code

Although only two HMD objects are used in this experiment, more than two objects can recognize each other by setting each object identifier.

Unity is updated frame by frame, and the packets used in this system are not transmitted to the middleware. Given the size of the transmitted data is not so large, the impact on the entire frame is subtle so that the user does not feel dizziness or a sense of awkwardness.

4-3 Experimental Process

1) Experimental Preparation

Based on the development contents, the following three experiments were conducted.

Table 3.

Subjects of Experiment

The frame per second (FPS) has the purpose of verifying how many frames are output per second in the HMD screen output. If the experimental result is low, the trainee feels dizziness and it is difficult to proceed with the training. The higher the measured value, the more natural the training becomes possible. Generally, 90FPS or more is used as a standard. The CPU operation delay rate measures the time it takes for the CPU to finish the 3D software operation and send the operation result to the GPU. The GPU output delay rate measures the time that the GPU actually outputs to the screen.

The measurement program used in each experiment is shown in Table 4 below.

Table 4.

Softwares Used in the Experiment

4-4 Result

The experimental results are shown in Fig 4 below.

Fig. 4.

Average Result Value

As a result of the experiment(Table 5), first, it can be confirmed that the delay rate is measured below the reference value when the content is executed through the middleware. The recommended level for FPS is over 90 fps and the average measured value is about 160 fps. The recommended level for CPU Operation Delay Time is less than 23 ms, and the average measured value is less than 5 ms. The recommended level for GPU Output Delay Time is less than 23ms and the average measured value is 6ms. According to the above experimental results, It can be seen that the delay rate is lower than the standard so that the middleware can synchronize in real time by providing common information that is not delayed to the trainees in controlling each client. Second, it does not affect the number of FPS. Since the number of FPS can be secured more than a certain value, this interactive mutually middleware system can confirm the position and motion of the other party on the own HMD even when the HMD is hermetically sealed.

Table 5.

Experiment Result Values(SET)