Korea Digital Contents Society
[ Article ]
Journal of Digital Contents Society - Vol. 25, No. 3, pp.595-605
ISSN: 1598-2009 (Print) 2287-738X (Online)
Print publication date 31 Mar 2024
Received 05 Jan 2024 Revised 13 Feb 2024 Accepted 04 Mar 2024
DOI: https://doi.org/10.9728/dcs.2024.25.3.595

Computer Vision-based Basketball Player Training System

Man Zhang1 ; Seung-Soon Shin2, *
1Ph.D.’s Course, Dept. of Computer and Media Engineering, Tongmyong University, Busan 48520, Korea
2Professor, Dept. of Information Security, Tongmyong University, Busan 48520, Korea
컴퓨터 비전 기반 농구선수 훈련 시스템
장만1 ; 신승수2, *
1동명대학교 컴퓨터미디어공학과 박사과정
2동명대학교 정보보호학과 교수

Correspondence to: *Seung-Soo Shin E-mail: shinsss@tu.ac.kr

Copyright ⓒ 2024 The Digital Contents Society
This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-CommercialLicense(http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Computer vision-based basketball player training system is a state-of-the-art field that relies on computer vision technologies and deep learning to help the basketball player train. Computer Vision-Based basketball player training systems rely on extensive training data to detect the player's movements and poses; the current training datasets are relatively small and vary in quality, affecting the basketball training effect. This paper proposes the human skeleton detection model, basketball and basketball hoop detection model, basketball shot detection model, basketball hit detection model, and basketball hit prediction model. In the proposed model, the basketball hit detection is improved by up to 9.66% over the existing method. The basketball hit prediction model is improved by up to 10.72% over the existing method using mathematical methods.

초록

컴퓨터 비전 기반 농구 선수 훈련 시스템은 컴퓨터 비전 기술과 딥러닝에 의존해 농구 선수의 훈련을 돕는 최첨단 분야이다. 컴퓨터 비전 기반의 농구 선수 훈련 시스템은 선수들의 움직임과 포즈를 감지하기 위해 광범위한 훈련 데이터에 의존하며, 현재 훈련 데이터 세트는 상대적으로 작고 품질이 다양하여 농구 훈련 효과에 많은 영향을 미친다. 본 논문에서는 인간 골격 감지 모델, 농구 및 농구 골대 감지 모델, 농구 숏 감지 모델, 농구 적중 감지 모델 및 농구 적중 예측 모델을 제안한다. 제안한 모델에서 농구 적중 감지는 기존 방식보다 최대 9.66% 향상되었다. 농구 적중 예측 모델의 경우, 수학적 방법을 사용하면 기존 방식보다 최대 10.72% 향상되었다.

Keywords:

Basketball Player Training, Object Detection, Convolutional Neural Network, Single Shot MultiBox Detector, Transfer Learning

키워드:

농구 선수 훈련, 객체 감지, 합성곱 신경망, 싱글숏 멀티박스 감지기, 전송 학습

Ⅰ. Introduction

Getting professional-level basketball guidance has been a challenge in many countries. In most professional basketball training centers, coaches are unable to offer personalized, one-on-one guidance to each player. Further, research indicates that a healthy adult can sustain a high-level of concentration for only about 10~15 minutes[1],[2]. Consequently, achieving ideal basketball training needs a significant commitment of time, energy, and money. In this paper, use computer vision technology to develop a cost-effective, efficient, and easy-to-implement basketball player training system[3].

This paper proposed a system to help basketball player to basketball training. This system mainly consists of the 5 models. Using the OpenPose and transfer learning method for the human skeleton detection model, using the Single Shot MultiBox Detector(SSD) algorithm and transfer learning for basketball and basketball hoop detection model, using the mathematics for basketball shot detection model, basketball hit detection and basketball hit prediction.

As computer vision technology turns out to be more and more mature, many researchers have utilized computer vision and related technologies applied to basketball player training[4]-[8]. Some researchers have proposed the use of computer vision technology to generate customized training plans based on each player’s different personal skill levels and training goals, which can greatly improve the skill level of basketball players[9],[10]. By using computer vision and related technologies, the function of real-time feedback can be realized[11].

The paper provides novel approaches to popularize and improve the sports of basketball, particularly for individuals who lack access to professional coaching. By combining computer vision and related techniques, this approach can also be applied to other sports to improve player performances[12].


Ⅱ. Related Works

2-1 Object Detection

Object detection is a computer vision task in which the goal is to detect and locate objects of interest in an image or video. The task involves identifying the position and boundaries of objects in an image and classifying the object into different categories. Felzenszwalb et.’s scheme[13] proposed an object detection algorithm called Deformable Part-based Model(DPM)in 2010. This algorithm is mainly based on the idea of component combination, which performs object detection by learning the components of an object and the spatial layout relationships between them.

2-2 OpenPose

OpenPose[14] is a real-time multi-person human pose detection library that has for the first time shown the capability to jointly detect the human body, foot, hand and facial key points on single images. OpenPose is capable of detecting a total of 135 key points. The OpenPose technique was developed at the robotics institute, Carnegie mellon university(CMU).

The overview pipeline of the OpenPose as shown in Figure 1, it consists of multiple sequential tasks: Acquisition of the entire image as input(image or video frame). Two-branch convolutional neural networks (CNNs) jointly predict confidence maps for body part detection, shown in (b). Estimate the part affinity field (PAF) for parts association, shown in (c). Set of bipartite matchings to associate body parts candidates. Assemble them into full-body poses for all people in the image.

Fig. 1.

Overview pipeline

The OpenPose takes as input a color image of size and produces, as output, an array of matrices which consists of the confidence maps of Key points and Part Affinity Heatmaps for each key point pair.

2-3 Single Shot MultiBox Detector

Single Shot MultiBox Detector[15],[16] were the most popular frameworks for object detection network because they could pinpoint the location of the multiple interested objects within an image and classify these found objects with confidences. In this paper, SSD framework with visual geometry group 16 layers (VGG16) backbone structure called SSD-VGG16 was considered to construct the basketball and basketball hoop detector.

2-4 Transfer Learning

Transfer learning[17] is the process of transferring the parameters of an already trained model(pre- trained model) to a new model to help train the new model. Transfer learning is a branch of deep learning that aims to use similarities between data, models to transfer knowledge learned in one domain to a new domain. Considering that most of the data or tasks are correlated, transfer learning allows us to share the parameters of the already learned model to the new model in a way that speeds up and optimizes the learning efficiency of the model and improves the accuracy.


Ⅲ. Basketball Player Training System

3-1 Design Philosophies

The basic design ideas of this system mainly include 5 models, as shown in Figure 2.

Fig. 2.

Overview of the proposed system

It includes human skeleton detection model, basketball and basketball hoop detection model, basketball shot detection model, basketball hit detection model, basketball hit prediction model.

3-2 Function Description

1) Human Skeleton Detection Model

For human skeleton detection model, dataset is built such as Carnegie Mellon University Panoptic Dataset, other model’s datasets are built by me, and label these datasets by myself. Using the OpenPose and transfer learning to train the human skeleton detection model, the human skeleton was detected, get the different human key point and the location of the each human key point.

2) Basketball and Basketball Hoop Detection Model

This model is detect the basketball and basketball hoop. Using the pre-trained VGG-16_SSD and transfer learning to train basketball and basketball hoop detection model, the basketball and basketball hoop was detected, get their location.

3) Basketball Shot Detection Model

For the basketball shot detection model, it is according to the human skeleton detection model, basketball and basketball hoop detection model, get the location of the hand and basketball, calculate their distance, to detect if the basketball is shot.

4) Basketball Hit Detection Model

For the basketball hit detection model, it is according to the human skeleton detection model, basketball, and basketball hoop detection model, get the location of the hand and basketball, to detect if basketball hits.

5) Basketball Hit Prediction Model

For the basketball hit prediction model, it is according to the human skeleton detection model, get the location of the shoulder, elbow, wrist, hip, knee, ankle, calculate the elbow angle and knee angle, then to predict if the basketball hits according to the database of exhaustive enumeration.

3-3 System Design

1) Human Skeleton Detection Model

This model is designed to detect the human key point. It mainly consists of dataset and Pre-trained Model offered by OpenPose, as shown in Figure 3.

Fig. 3.

Flowchart of the human skeleton detection

Pre-trained Model is based on Deep neural network and human key point detection technologies. The basic principle is to use CNN to extract Part Confidence Maps and Part Affinity Fields, which help to locate the position of key points, and then use Bipartite Matching of graph theory to derive Part Association, which connects the key point of the same person. We used the CMU Panoptic Dataset to train human skeleton detection model in this paper. During the detection process, the 18 key point in the picture or video are classified, such as nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear, left ear.

2) Basketball and Basketball Hoop Detection Model

This model is designed to detect the basketball and basketball hoop. It mainly consists of dataset, pre- trained SSD_VGG16 model and transfer learning. The principle of SSD_VGG16 is to use the pre-trained weights of VGG16 as a feature extractor in the SSD model, and then add additional convolutional and prediction layers on top of it to accomplish the object detection, these additional layers are used to perform multi-scale regression of the object location and classification prediction of the object class on top of the extracted features, as shown in Figure 4.

Fig. 4.

Flowchart of the basketball and basketball hoop detection

3) Basketball Shot Detection Model

This model is based on recognizing if the basketball is out of the hand to detect basketball shot, as shown in Figure 5.

Fig. 5.

Flowchart of the basketball shot detection

According to the human skeleton detection model, the location of the key point of the hand(at the right and left wrists) can be get, as A. Based on the basketball and basketball hoop detection model, get the location of the basketball, as B. Calculate the distance of the hand and the basketball by the following formula:

d=B-A(1) 

where, d is the value of the distance between the hand and the basketball.

If d is bigger than threshold, the basketball can be recognized to be out of the hand, it means basketball shot. According to the actual situation, consider the distance between the hand and the basketball ball and other related features to go to set the threshold value. The threshold value is decided by the diameter of the basketball and distance between the characters captured on the camera.

4) Basketball Hit Detection Model

This model is designed to detect whether basketball hit or not. The flow chart of the basketball hit detection model is shown as in Figure 6.

Fig. 6.

Flowchart of the basketball hit detection

The principle of this model is as follow:

a. Calculate Slope: According to the basketball and basketball hoop detection model to get the coordinated of the basketball and basketball hoop. Calculate the slope between the known coordinates of the basketball and the hoop. This slope represents the direction of the basketball shot.

b. Construct Perpendicular Line: Derive the slope perpendicular to the shot trajectory by taking the negative reciprocal of the known slope. This process relies on the geometric relationship between perpendicular lines.

c. Build Line Equation: Utilize the coordinates of the hoop as known points to construct an equation for a line perpendicular to the shot trajectory. This equation describes a line perpendicular to the direction of the basketball shot.

d. Hit Detection: Finally, assess whether the basketball’s coordinates satisfy the equation of this perpendicular line. If the basketball’s coordinates align with this line equation, it signifies an intersection between the basketball’s path and the constructed perpendicular line, indicating a successful shot. Conversely, if there is no alignment, it suggests that the basketball’s path does not intersect with this line, indicating basketball shot not.

5) Basketball Hit Prediction Model

This model is designed to predict whether basketball shot will hit or not. The player’s height range(160~210cm), as well as shot speed and shot angle are calculated iteratively by exhaustive enumeration, and the shot speed and shot angle are stored in the database along with the corresponding results. When we get the player’s shot speed and shot angle, according to the data from database, we can quickly predict whether the player can hit or not.

For calculating the elbow angle and knee angle, first, according to the human skeleton detection model, recognize the key point of the shoulder, then output the location, denoted as S. Recognize the key point of the elbow, then output the location, denoted as S. Recognize the key point of the elbow, then output the location, denoted as E. Recognize the key point of the wrist, then output the location, denoted as W.

Recognize the key point at the hip, then output the location, noted as H. Recognize the key point of the knee, then output the location, noted as K. Recognize the key point at the ankle, then output the location, noted as A.

Calculate the elbow angle θelbow based on the location of the key point at the shoulder, elbow, and wrist with the following formula:

θelbow =arccosSE×EWSE×EW(2) 

Calculate the knee angle θknee based on the location of the key point at the hip, knee, and ankle with the following formula:

θknee =arccosHK×KAHK×KA(3) 

To convert radians to degrees by using the following formula:

Degrees = Radians ×180π(4) 

Where, Degrees is the angle in degrees, Radians is the angle in radians, π is the mathematical constant pi, approximately equal to 3.14159.


Ⅳ. Evaluation

The performance evaluation of the proposed model is shown accuracy, confusion matrix, precision, recall, F1-Score and support with Liu et al.’s scheme[15] and Quijano et al.’s scheme[17].

4-1 Experiments

1) Dataset Collection

In this paper, for human skeleton detection model, this paper uses the dataset from website of the CMU(Carnegie Mellon University) Panoptic Dataset. For other models, we collected the dataset by using the camera and labeled by labelme tool. The details of dataset collection as shown in Table 1.

Dataset collection for 5 models

2) Dataset Labeling

In this paper, for model training, it needs label the dataset. We label the dataset by myself and for the details is shown in Table 2.

Label classes of dataset for proposed models

3) Datasets Training

In this paper, use the hardware and software to train and test the model. The training and testing environment is shown in Table 3, for each model, use the different method to train and test, the details are shown in Table 4. Each module was tuned and trained with ResNet50, transfer learning and hyperparameters according to Table 4.

Testing environments

Method of proposed models

4-2 Performance Evaluation

In this paper, we use the accuracy, confusion matrix, precision, recall, F1-Score and support as the performance matrix for each model.

1) Human Skeleton Detection Model

In this section, we describe the results for human skeleton detection model. The label class name is shown in Table 2. Confusion matrix is a performance measurement for the object detection where the output can be two or more labels. It is a table with combinations of predicted and actual labels. A confusion matrix defined as the table that is often used to describe the performance of a classification model on a set of the test data for which the true values are known. Confusion matrix of human skeleton detection is shown in Figure 7, performance of accuracy, precision, recall, F1-Score, support is shown in Table 5 and the result image of human skeleton detection is shown in Figure 8.

Fig. 7.

Confusion matrix of human skeleton detection

Performance of human skeleton detection

Fig. 8.

Result image of human skeleton detection

2) Basketball and Basketball Hoop Detection Model

In this section, we describe the results for basketball and basketball hoop detection model. Confusion matrix is shown in Figure 9, performance of accuracy, precision, recall, F1-Score, support is shown in Table 6 and the result image is shown in Figure 10.

Fig. 9.

Confusion matrix of basketball and basketball hoop detection

Performance of basketball and basketball hoop detection

Fig. 10.

Result image of basketball and basketball hoop detection

3) Basketball Shot Detection Model

In this section, we describe the results for basketball shot detection model. Confusion matrix is shown in Figure 11, performance of accuracy, precision, recall, F1-Score, support is shown in Table 7 and the result image is shown in Figure 12.

Fig. 11.

Confusion matrix of basketball shot detection

Performance of basketball shot detection

Fig. 12.

Result image of basketball shot detection

4) Basketball Hit Detection Model

In this section, we describe the results for basketball hit detection model. Confusion Matrix is shown in Figure 13, performance of accuracy, precision, recall, F1-Score, support is shown in Table 8 and the result image is shown in Figure 14.

Fig. 13.

Confusion matrix of basketball hit detection

Performance of basketball hit detection

Fig. 14.

Result image of basketball hit detection

5) Basketball Hit Prediction Model

In this section, we describe the results for basketball hit prediction model. Confusion Matrix is shown in Figure 15, performance of accuracy, precision, recall, F1-Score, support is shown in Table 9 and the result image is shown in Figure 16.

Fig. 15.

Confusion matrix of basketball hit prediction

Performance of basketball hit prediction

Fig. 16.

Result image of basketball hit prediction

4-3 Analysis

1) Human Skeleton Detection Model

For the human skeleton detection model, model performance of this paper, Liu et al.’s scheme[15] and Quijano et al.’s scheme[17] as shown in Table 10. Compared with Liu et al.’s scheme[15], the accuracy of this paper is higher 13%. Compared with Quijano et al.’s scheme[17], the accuracy of this paper is higher 8%.

Performance of human skeleton detection

2) Basketball and Basketball Hoop Detection Model

For the basketball and basketball hoop detection model, model performance of this paper, Liu et al.’s scheme[15] and Quijano et al.’s scheme[17] as shown in Table 11. Compared with Liu et al.’s scheme[15], the accuracy of this paper is higher 7%. Compared with Quijano et al.’s scheme[17], the accuracy of this paper is higher 8%.

Performance of basketball and basketball hoop detection

3) Basketball Shot Detection Model

For the basketball shot detection model, model performance of this paper, Liu et al.’s scheme[15] and Quijano et al.’s scheme[17] as shown in Table 12. Compared with Liu et al.’s scheme[15], the accuracy of this paper is higher 9.76%. Compared with Quijano et al.’s scheme[17], the accuracy of this paper is higher 7.76%.

Performance of basketball shot detection

4) Basketball Hit Detection Model

For the basketball hit detection model, model performance of this paper, Liu et al.’s scheme[15] and Quijano et al’s scheme[17] as shown in Table 13. Compared with Liu et al.’s scheme[15], the accuracy of this paper is higher 9.66%. Compared with Quijano et al.’s scheme[17], the accuracy of this paper is higher 6.76%.

Performance of basketball hit detection

5) Basketball Hit Prediction Model

For the basketball hit prediction model performance of this paper, Liu et al.’s scheme[15] and Quijano et al.’s scheme[17] as shown in Table 14. For basketball hit prediction model, compared with Liu et al.’s scheme[15], the accuracy of this paper is higher 10.72%. Compared with Quijano et al.’s scheme[17], the accuracy of this paper is higher 5.72%.

Performance of basketball hit prediction


Ⅴ. Conclusion

This paper presents a comprehensive computer vision-based basketball player training system. By comparing the accuracy of different methods, the system achieves significant improvements.

For the human skeleton detection model, Utilizing the OpenPose and transfer learning method for human skeleton detection results in a 13% improvement over Liu et al.’s scheme[15] and an 8% improvement over Quijano et al.’s scheme[17].

For the basketball and basketball hoop detection model, using the SSD algorithm and transfer learning for basketball and basketball hoop detection achieves a 7% improvement over Liu et al.’s scheme[15] and an 8% improvement over Quijano et al.’s scheme[17].

For the basketball shot detection model, using mathematical methods results in 9.76% improvements over Liu et al.’s scheme[15] and 7.76% over Quijano et al.’s scheme[17].

For the basketball hit detection model, using mathematical methods results in 9.66% improvements over Liu et al.’s scheme[15] and 6.76% over Quijano et al.’s scheme[17].

For the basketball hit prediction model, using mathematical methods results in 10.72% improvements over Liu et al.’s scheme[15] and 5.72% over Quijano et al.’s scheme[17].

Enhancing the system’s real-time performance and stability to enable efficient operation in practical training scenarios. Exploring more data augmentation techniques and model optimization methods to improve the model’s generalization and robustness. Further advancements and applications of the computer vision-based basketball player training system can be developed and promoted.

References

  • B. G. Davis, Tools for Teaching, San Francisco, CA: Jossey-Bass, 1993.
  • P. C. Wankat, The Effective, Efficient Professor: Teaching, Scholarship and Service, Boston, MA: Allyn and Bacon, 2002.
  • Z. Liu, “Application of Artificial Intelligence Technology in Basketball Games,” IOP Conference Series: Materials Science and Engineering, Vol. 750, 012093. [https://doi.org/10.1088/1757-899X/750/1/012093]
  • W. Yan, X. Jiang, and P. Liu, “A Review of Basketball Shooting Analysis Based on Artificial Intelligence,” IEEE Access, Vol. 11, pp. 87344-87365, 2023. [https://doi.org/10.1109/ACCESS.2023.3304631]
  • H. Meng, “Deep Learning Algorithm and Video Image Processing-Based Basketball Training System,” in Proceedings of 2023 International Conference on Applied Intelligence and Sustainable Computing (ICAISC), Dharwad, India, pp. 1-6, June 2023. [https://doi.org/10.1109/ICAISC58445.2023.10200635]
  • J. Zhou, “Automatic Detection Approach of Basketball Track in Sports Video,” in Proceedings of 2021 International Conference on Intelligent Transportation, Big Data & Smart City (ICITBS), Xi’an, China, pp. 630-633, March 2021. [https://doi.org/10.1109/ICITBS53129.2021.00160]
  • R. Ji, “Research on Basketball Shooting Action Based on Image Feature Extraction and Machine Learning,” IEEE Access, Vol. 8, pp. 138743-138751, 2020. [https://doi.org/10.1109/ACCESS.2020.3012456]
  • W.-L. Tsai, L.-W. Su, T.-Y. Ko, T.-Y. Pan, and M.-C. Hu, “Feasibility Study on Using AI and VR for Decision-Making Training of Basketball Players,” IEEE Transactions on Learning Technologies, Vol. 14, No. 6, pp. 754-762, December 2021. [https://doi.org/10.1109/TLT.2022.3145093]
  • I. Sassi, S. Anter, and A. Bekkhoucha, “A New Improved Baum-Welch Algorithm for Unsupervised Learning for Continuous-Time HMM Using Spark,” International Journal of Intelligent Engineering and Systems, Vol. 13, No. 1, pp. 214-226, 2020. [https://doi.org/10.22266/ijies2020.0229.20]
  • E. Epaillard and N. Bouguila, “Variational Bayesian Learning of Generalized Dirichlet-Based Hidden Markov Models Applied to Unusual Events Detection,” IEEE Transactions on Neural Networks and Learning Systems, Vol. 30, No. 4, pp. 1034-1047, April 2019. [https://doi.org/10.1109/TNNLS.2018.2855699]
  • Z. Li and D. Yu, “Application Prospect of Artificial Intelligence Technology in Vestibular Disorders,” Journal of Clinical Otorhinolaryngology Head and Neck Surgery, Vol. 33, No. 9, pp. 895-897, 2019. [https://doi.org/10.13201/j.issn.1001-1781.2019.09.025]
  • B. Li and X. Xu, “Application of Artificial Intelligence in Basketball Sport,” Journal of Education, Health and Sport, Vol. 11, No. 7, pp. 54-67, 2021. [https://doi.org/10.12775/JEHS.2021.11.07.005]
  • P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan, “Object Detection with Discriminatively Trained Part-Based Models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, pp. 1627-1645, September 2010. [https://doi.org/10.1109/TPAMI.2009.167]
  • Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y. Sheikh, “OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 43, No. 1, pp. 172-186, January 2021. [https://doi.org/10.1109/TPAMI.2019.2929257]
  • W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “SSD: Single Shot Multibox Detector,” in Proceedings of the 14th European Conference on Computer Vision (ECCV 2016), Amsterdam, Netherlands, pp. 21-37, October 2016. [https://doi.org/10.1007/978-3-319-46448-0_2]
  • C. Loraksa, S. Mongkolsomlit, N. Nimsuk, M. Uscharapong, and P. Kiatisevi, “Development of the Osteosarcoma Lung Nodules Detection Model Based on SSD-VGG16 and Competency Comparing With Traditional Method,” IEEE Access, Vol. 10, pp. 65496-65506, 2022. [https://doi.org/10.1109/ACCESS.2022.3183604]
  • W. Liu, K. Quijano, and M. M. Crawford, “YOLOv5-Tassel: Detecting Tassels in RGB UAV Imagery With Improved YOLOv5 Based on Transfer Learning,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 15, pp. 8085-8094, 2022. [https://doi.org/10.1109/JSTARS.2022.3206399]

저자소개

장 만(Man Zhang)

2017년 7월:Henan University (Bachelor)

2019년 5월:Pace University (Master)

2022년 3월~현 재: 동명대학교 컴퓨터미디어공학과 박사과정

※관심분야:AI, 딥러닝, 빅테이터, CNN

신승수(Seung-Soo Shin)

2001년 2월:충북대학교 수학과 (이학박사)

2004년 8월:충북대학교 컴퓨터공학과 (공학박사)

2005년 3월~현 재: 동명대학교 정보보호학과 교수

※관심분야:네트워크 보안, 딥러닝, IoT, 데이터분석

Fig. 1.

Fig. 1.
Overview pipeline

Fig. 2.

Fig. 2.
Overview of the proposed system

Fig. 3.

Fig. 3.
Flowchart of the human skeleton detection

Fig. 4.

Fig. 4.
Flowchart of the basketball and basketball hoop detection

Fig. 5.

Fig. 5.
Flowchart of the basketball shot detection

Fig. 6.

Fig. 6.
Flowchart of the basketball hit detection

Fig. 7.

Fig. 7.
Confusion matrix of human skeleton detection

Fig. 8.

Fig. 8.
Result image of human skeleton detection

Fig. 9.

Fig. 9.
Confusion matrix of basketball and basketball hoop detection

Fig. 10.

Fig. 10.
Result image of basketball and basketball hoop detection

Fig. 11.

Fig. 11.
Confusion matrix of basketball shot detection

Fig. 12.

Fig. 12.
Result image of basketball shot detection

Fig. 13.

Fig. 13.
Confusion matrix of basketball hit detection

Fig. 14.

Fig. 14.
Result image of basketball hit detection

Fig. 15.

Fig. 15.
Confusion matrix of basketball hit prediction

Fig. 16.

Fig. 16.
Result image of basketball hit prediction

Table 1.

Dataset collection for 5 models

Dataset for Models Training Data Sizes Testing Data Sizes
Human Skeleton Detection 21,600 21,600
Basketball and Basketball Hoop Detection 5,000 5,000
Basketball Shot Detection 5,000 5,000
Basketball Hit Detection 5,000 5,000
Basketball Hit Prediction 4,950 4,950

Table 2.

Label classes of dataset for proposed models

Dataset for Models Label Class
Human Skeleton Detection 0: Nose
2: Right Shoulder
4: Right Wrist
6: Left Elbow
8: Right Hip
10: Right Ankle
12: Left Knee
14: Right Eye
16: Right Ear
1: Neck
3: Right Elbow
5: Left Shoulder
7: Left Wrist
9: Right Knee
11: Left Hip
13: Left Ankle
15: Left Eye
17: Left Ear
Basketball and Basketball Hoop Detection 0: Basketball
1: Basketball Hoop
2: Background
Basketball Shot Detection 0: Not     1: Shot
Basketball Hit Detection 0: Miss    1: Make
Basketball Hit Prediction 0: Miss    1: Make

Table 3.

Testing environments

Resources Specification
CPU ⦁EPYC 9004×2
Language ⦁A100×4
OS ⦁Windows Server 2022   ⦁Ubuntu 22.04
Libraries ⦁Python 3.8
⦁Tensorflow 1.15.0
⦁OpenCV
⦁GCC 8.5
⦁Pytorch 2.0
⦁OpenPose

Table 4.

Method of proposed models

Proposed Models Methods
Human Skeleton Detection OpenPose+Transfer Learning
Basketball and Basketball Hoop Detection SSD+Transfer Learning
Basketball Shot Detection Mathematics
Basketball Hit Detection Mathematics
Basketball Hit Prediction Mathematics

Table 5.

Performance of human skeleton detection

Label Class Accuracy Precision Recall F1-Score Support
0 0.94 0.44 0.67 0.53 1,200
1 1.00 1.00 0.92 0.96 1,200
2 1.00 1.00 1.00 1.00 1,200
3 0.95 0.60 0.50 0.55 1,200
4 0.95 0.55 0.50 0.52 1,200
5 0.98 0.89 0.67 0.76 1,200
6 0.96 0.57 1.00 0.73 1,200
7 0.98 0.75 1.00 0.86 1,200
8 0.99 1.00 0.83 0.91 1,200
9 1.00 1.00 1.00 1.00 1,200
10 0.95 0.60 0.25 0.35 1,200
11 1.00 1.00 1.00 1.00 1,200
12 1.00 1.00 1.00 1.00 1,200
13 0.98 0.75 1.00 0.86 1,200
14 0.99 0.80 1.00 0.89 1,200
15 0.96 0.83 0.42 0.56 1,200
16 0.98 0.89 0.67 0.76 1,200
17 0.96 0.64 0.58 0.61 1,200
Total 0.98 0.79 0.78 0.77 21,600

Table 6.

Performance of basketball and basketball hoop detection

Label Class Accuracy Precision Recall F1-Score Support
0 0.99 0.99 1.00 0.99 2,000
1 0.99 1.00 0.99 0.99 2,000
2 1.00 1.00 1.00 1.00 1.000
Total 0.99 0.99 1.00 0.99 5,000

Table 7.

Performance of basketball shot detection

Label Class Accuracy Precision Recall F1-Score Support
0 1.00 1.00 0.98 0.99 500
1 1.00 1.00 1.00 1.00 4,500
Total 0.9976 1.00 0.99 0.99 5,000

Table 8.

Performance of basketball hit detection

Label Class Accuracy Precision Recall F1-Score Support
0 0.98 0.87 0.90 0.88 500
1 0.98 0.99 0.98 0.99 4,500
Total 0.9760 0.93 0.94 0.93 5,000

Table 9.

Performance of basketball hit prediction

Label Class Accuracy Precision Recall F1-Score Support
0 0.98 0.96 1.00 0.98 2,500
1 0.98 1.00 0.96 0.98 2,450
Total 0.9772 0.98 0.98 0.98 4,950

Table 10.

Performance of human skeleton detection

Model Scheme Method Accuracy
Human Skeleton Detection Model Proposed Scheme OpenPose + Transfer Learning 98%
Liu et al.'s Scheme[15] SSD 85%
Quijano et al.'s Scheme[17] YOLO 90%

Table 11.

Performance of basketball and basketball hoop detection

Model Scheme Method Accuracy
Basketball and Basketball Hoop Detection Model Proposed Scheme SSD + Transfer Learning 99%
Liu et al.'s Scheme[15] SSD 92%
Quijano et al.'s Scheme[17] YOLO 91%

Table 12.

Performance of basketball shot detection

Model Scheme Method Accuracy
Basketball Shot Detection Model Proposed Scheme Mathematic 99.76%
Liu et al.'s Scheme[15] SSD 90%
Quijano et al.'s Scheme[17] YOLO 92%

Table 13.

Performance of basketball hit detection

Model Scheme Method Accuracy
Basketball Hit Detection Model Proposed Scheme Mathematic 97.6%
Liu et al.'s Scheme[15] SSD 88%
Quijano et al.'s Scheme[17] YOLO 91%

Table 14.

Performance of basketball hit prediction

Model Scheme Method Accuracy
Basketball Hit Prediction Model Proposed Scheme Mathematic 97.72%
Liu et al.'s Scheme[15] SSD 87%
Quijano et al.'s Scheme[17] YOLO 92%