Computer Vision-based Basketball Player Training System
Copyright ⓒ 2024 The Digital Contents Society
This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-CommercialLicense(http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
Abstract
Computer vision-based basketball player training system is a state-of-the-art field that relies on computer vision technologies and deep learning to help the basketball player train. Computer Vision-Based basketball player training systems rely on extensive training data to detect the player's movements and poses; the current training datasets are relatively small and vary in quality, affecting the basketball training effect. This paper proposes the human skeleton detection model, basketball and basketball hoop detection model, basketball shot detection model, basketball hit detection model, and basketball hit prediction model. In the proposed model, the basketball hit detection is improved by up to 9.66% over the existing method. The basketball hit prediction model is improved by up to 10.72% over the existing method using mathematical methods.
초록
컴퓨터 비전 기반 농구 선수 훈련 시스템은 컴퓨터 비전 기술과 딥러닝에 의존해 농구 선수의 훈련을 돕는 최첨단 분야이다. 컴퓨터 비전 기반의 농구 선수 훈련 시스템은 선수들의 움직임과 포즈를 감지하기 위해 광범위한 훈련 데이터에 의존하며, 현재 훈련 데이터 세트는 상대적으로 작고 품질이 다양하여 농구 훈련 효과에 많은 영향을 미친다. 본 논문에서는 인간 골격 감지 모델, 농구 및 농구 골대 감지 모델, 농구 숏 감지 모델, 농구 적중 감지 모델 및 농구 적중 예측 모델을 제안한다. 제안한 모델에서 농구 적중 감지는 기존 방식보다 최대 9.66% 향상되었다. 농구 적중 예측 모델의 경우, 수학적 방법을 사용하면 기존 방식보다 최대 10.72% 향상되었다.
Keywords:
Basketball Player Training, Object Detection, Convolutional Neural Network, Single Shot MultiBox Detector, Transfer Learning키워드:
농구 선수 훈련, 객체 감지, 합성곱 신경망, 싱글숏 멀티박스 감지기, 전송 학습Ⅰ. Introduction
Getting professional-level basketball guidance has been a challenge in many countries. In most professional basketball training centers, coaches are unable to offer personalized, one-on-one guidance to each player. Further, research indicates that a healthy adult can sustain a high-level of concentration for only about 10~15 minutes[1],[2]. Consequently, achieving ideal basketball training needs a significant commitment of time, energy, and money. In this paper, use computer vision technology to develop a cost-effective, efficient, and easy-to-implement basketball player training system[3].
This paper proposed a system to help basketball player to basketball training. This system mainly consists of the 5 models. Using the OpenPose and transfer learning method for the human skeleton detection model, using the Single Shot MultiBox Detector(SSD) algorithm and transfer learning for basketball and basketball hoop detection model, using the mathematics for basketball shot detection model, basketball hit detection and basketball hit prediction.
As computer vision technology turns out to be more and more mature, many researchers have utilized computer vision and related technologies applied to basketball player training[4]-[8]. Some researchers have proposed the use of computer vision technology to generate customized training plans based on each player’s different personal skill levels and training goals, which can greatly improve the skill level of basketball players[9],[10]. By using computer vision and related technologies, the function of real-time feedback can be realized[11].
The paper provides novel approaches to popularize and improve the sports of basketball, particularly for individuals who lack access to professional coaching. By combining computer vision and related techniques, this approach can also be applied to other sports to improve player performances[12].
Ⅱ. Related Works
2-1 Object Detection
Object detection is a computer vision task in which the goal is to detect and locate objects of interest in an image or video. The task involves identifying the position and boundaries of objects in an image and classifying the object into different categories. Felzenszwalb et.’s scheme[13] proposed an object detection algorithm called Deformable Part-based Model(DPM）in 2010. This algorithm is mainly based on the idea of component combination, which performs object detection by learning the components of an object and the spatial layout relationships between them.
2-2 OpenPose
OpenPose[14] is a real-time multi-person human pose detection library that has for the first time shown the capability to jointly detect the human body, foot, hand and facial key points on single images. OpenPose is capable of detecting a total of 135 key points. The OpenPose technique was developed at the robotics institute, Carnegie mellon university(CMU).
The overview pipeline of the OpenPose as shown in Figure 1, it consists of multiple sequential tasks: Acquisition of the entire image as input(image or video frame). Two-branch convolutional neural networks (CNNs) jointly predict confidence maps for body part detection, shown in (b). Estimate the part affinity field (PAF) for parts association, shown in (c). Set of bipartite matchings to associate body parts candidates. Assemble them into full-body poses for all people in the image.
Overview pipeline
The OpenPose takes as input a color image of size and produces, as output, an array of matrices which consists of the confidence maps of Key points and Part Affinity Heatmaps for each key point pair.
2-3 Single Shot MultiBox Detector
Single Shot MultiBox Detector[15],[16] were the most popular frameworks for object detection network because they could pinpoint the location of the multiple interested objects within an image and classify these found objects with confidences. In this paper, SSD framework with visual geometry group 16 layers (VGG16) backbone structure called SSD-VGG16 was considered to construct the basketball and basketball hoop detector.
2-4 Transfer Learning
Transfer learning[17] is the process of transferring the parameters of an already trained model(pre- trained model) to a new model to help train the new model. Transfer learning is a branch of deep learning that aims to use similarities between data, models to transfer knowledge learned in one domain to a new domain. Considering that most of the data or tasks are correlated, transfer learning allows us to share the parameters of the already learned model to the new model in a way that speeds up and optimizes the learning efficiency of the model and improves the accuracy.
Ⅲ. Basketball Player Training System
3-1 Design Philosophies
The basic design ideas of this system mainly include 5 models, as shown in Figure 2.
Overview of the proposed system
It includes human skeleton detection model, basketball and basketball hoop detection model, basketball shot detection model, basketball hit detection model, basketball hit prediction model.
3-2 Function Description
For human skeleton detection model, dataset is built such as Carnegie Mellon University Panoptic Dataset, other model’s datasets are built by me, and label these datasets by myself. Using the OpenPose and transfer learning to train the human skeleton detection model, the human skeleton was detected, get the different human key point and the location of the each human key point.
This model is detect the basketball and basketball hoop. Using the pre-trained VGG-16_SSD and transfer learning to train basketball and basketball hoop detection model, the basketball and basketball hoop was detected, get their location.
For the basketball shot detection model, it is according to the human skeleton detection model, basketball and basketball hoop detection model, get the location of the hand and basketball, calculate their distance, to detect if the basketball is shot.
For the basketball hit detection model, it is according to the human skeleton detection model, basketball, and basketball hoop detection model, get the location of the hand and basketball, to detect if basketball hits.
For the basketball hit prediction model, it is according to the human skeleton detection model, get the location of the shoulder, elbow, wrist, hip, knee, ankle, calculate the elbow angle and knee angle, then to predict if the basketball hits according to the database of exhaustive enumeration.
3-3 System Design
This model is designed to detect the human key point. It mainly consists of dataset and Pre-trained Model offered by OpenPose, as shown in Figure 3.
Flowchart of the human skeleton detection
Pre-trained Model is based on Deep neural network and human key point detection technologies. The basic principle is to use CNN to extract Part Confidence Maps and Part Affinity Fields, which help to locate the position of key points, and then use Bipartite Matching of graph theory to derive Part Association, which connects the key point of the same person. We used the CMU Panoptic Dataset to train human skeleton detection model in this paper. During the detection process, the 18 key point in the picture or video are classified, such as nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear, left ear.
This model is designed to detect the basketball and basketball hoop. It mainly consists of dataset, pre- trained SSD_VGG16 model and transfer learning. The principle of SSD_VGG16 is to use the pre-trained weights of VGG16 as a feature extractor in the SSD model, and then add additional convolutional and prediction layers on top of it to accomplish the object detection, these additional layers are used to perform multi-scale regression of the object location and classification prediction of the object class on top of the extracted features, as shown in Figure 4.
Flowchart of the basketball and basketball hoop detection
This model is based on recognizing if the basketball is out of the hand to detect basketball shot, as shown in Figure 5.
Flowchart of the basketball shot detection
According to the human skeleton detection model, the location of the key point of the hand(at the right and left wrists) can be get, as A. Based on the basketball and basketball hoop detection model, get the location of the basketball, as B. Calculate the distance of the hand and the basketball by the following formula:
| (1) |
where, d is the value of the distance between the hand and the basketball.
If d is bigger than threshold, the basketball can be recognized to be out of the hand, it means basketball shot. According to the actual situation, consider the distance between the hand and the basketball ball and other related features to go to set the threshold value. The threshold value is decided by the diameter of the basketball and distance between the characters captured on the camera.
This model is designed to detect whether basketball hit or not. The flow chart of the basketball hit detection model is shown as in Figure 6.
Flowchart of the basketball hit detection
The principle of this model is as follow:
a. Calculate Slope: According to the basketball and basketball hoop detection model to get the coordinated of the basketball and basketball hoop. Calculate the slope between the known coordinates of the basketball and the hoop. This slope represents the direction of the basketball shot.
b. Construct Perpendicular Line: Derive the slope perpendicular to the shot trajectory by taking the negative reciprocal of the known slope. This process relies on the geometric relationship between perpendicular lines.
c. Build Line Equation: Utilize the coordinates of the hoop as known points to construct an equation for a line perpendicular to the shot trajectory. This equation describes a line perpendicular to the direction of the basketball shot.
d. Hit Detection: Finally, assess whether the basketball’s coordinates satisfy the equation of this perpendicular line. If the basketball’s coordinates align with this line equation, it signifies an intersection between the basketball’s path and the constructed perpendicular line, indicating a successful shot. Conversely, if there is no alignment, it suggests that the basketball’s path does not intersect with this line, indicating basketball shot not.
This model is designed to predict whether basketball shot will hit or not. The player’s height range(160~210cm), as well as shot speed and shot angle are calculated iteratively by exhaustive enumeration, and the shot speed and shot angle are stored in the database along with the corresponding results. When we get the player’s shot speed and shot angle, according to the data from database, we can quickly predict whether the player can hit or not.
For calculating the elbow angle and knee angle, first, according to the human skeleton detection model, recognize the key point of the shoulder, then output the location, denoted as S. Recognize the key point of the elbow, then output the location, denoted as S. Recognize the key point of the elbow, then output the location, denoted as E. Recognize the key point of the wrist, then output the location, denoted as W.
Recognize the key point at the hip, then output the location, noted as H. Recognize the key point of the knee, then output the location, noted as K. Recognize the key point at the ankle, then output the location, noted as A.
Calculate the elbow angle θelbow based on the location of the key point at the shoulder, elbow, and wrist with the following formula:
| (2) |
Calculate the knee angle θknee based on the location of the key point at the hip, knee, and ankle with the following formula:
| (3) |
To convert radians to degrees by using the following formula:
| (4) |
Where, Degrees is the angle in degrees, Radians is the angle in radians, π is the mathematical constant pi, approximately equal to 3.14159.
Ⅳ. Evaluation
The performance evaluation of the proposed model is shown accuracy, confusion matrix, precision, recall, F1-Score and support with Liu et al.’s scheme[15] and Quijano et al.’s scheme[17].
4-1 Experiments
In this paper, for human skeleton detection model, this paper uses the dataset from website of the CMU(Carnegie Mellon University) Panoptic Dataset. For other models, we collected the dataset by using the camera and labeled by labelme tool. The details of dataset collection as shown in Table 1.
Dataset collection for 5 models
In this paper, for model training, it needs label the dataset. We label the dataset by myself and for the details is shown in Table 2.
Label classes of dataset for proposed models
In this paper, use the hardware and software to train and test the model. The training and testing environment is shown in Table 3, for each model, use the different method to train and test, the details are shown in Table 4. Each module was tuned and trained with ResNet50, transfer learning and hyperparameters according to Table 4.
Testing environments
Method of proposed models
4-2 Performance Evaluation
In this paper, we use the accuracy, confusion matrix, precision, recall, F1-Score and support as the performance matrix for each model.
In this section, we describe the results for human skeleton detection model. The label class name is shown in Table 2. Confusion matrix is a performance measurement for the object detection where the output can be two or more labels. It is a table with combinations of predicted and actual labels. A confusion matrix defined as the table that is often used to describe the performance of a classification model on a set of the test data for which the true values are known. Confusion matrix of human skeleton detection is shown in Figure 7, performance of accuracy, precision, recall, F1-Score, support is shown in Table 5 and the result image of human skeleton detection is shown in Figure 8.
Confusion matrix of human skeleton detection
Performance of human skeleton detection
Result image of human skeleton detection
In this section, we describe the results for basketball and basketball hoop detection model. Confusion matrix is shown in Figure 9, performance of accuracy, precision, recall, F1-Score, support is shown in Table 6 and the result image is shown in Figure 10.
Confusion matrix of basketball and basketball hoop detection
Performance of basketball and basketball hoop detection
Result image of basketball and basketball hoop detection
In this section, we describe the results for basketball shot detection model. Confusion matrix is shown in Figure 11, performance of accuracy, precision, recall, F1-Score, support is shown in Table 7 and the result image is shown in Figure 12.
Confusion matrix of basketball shot detection
Performance of basketball shot detection
Result image of basketball shot detection
In this section, we describe the results for basketball hit detection model. Confusion Matrix is shown in Figure 13, performance of accuracy, precision, recall, F1-Score, support is shown in Table 8 and the result image is shown in Figure 14.
Confusion matrix of basketball hit detection
Performance of basketball hit detection
Result image of basketball hit detection
In this section, we describe the results for basketball hit prediction model. Confusion Matrix is shown in Figure 15, performance of accuracy, precision, recall, F1-Score, support is shown in Table 9 and the result image is shown in Figure 16.
Confusion matrix of basketball hit prediction
Performance of basketball hit prediction
Result image of basketball hit prediction
4-3 Analysis
For the human skeleton detection model, model performance of this paper, Liu et al.’s scheme[15] and Quijano et al.’s scheme[17] as shown in Table 10. Compared with Liu et al.’s scheme[15], the accuracy of this paper is higher 13%. Compared with Quijano et al.’s scheme[17], the accuracy of this paper is higher 8%.
Performance of human skeleton detection
For the basketball and basketball hoop detection model, model performance of this paper, Liu et al.’s scheme[15] and Quijano et al.’s scheme[17] as shown in Table 11. Compared with Liu et al.’s scheme[15], the accuracy of this paper is higher 7%. Compared with Quijano et al.’s scheme[17], the accuracy of this paper is higher 8%.
Performance of basketball and basketball hoop detection
For the basketball shot detection model, model performance of this paper, Liu et al.’s scheme[15] and Quijano et al.’s scheme[17] as shown in Table 12. Compared with Liu et al.’s scheme[15], the accuracy of this paper is higher 9.76%. Compared with Quijano et al.’s scheme[17], the accuracy of this paper is higher 7.76%.
Performance of basketball shot detection
For the basketball hit detection model, model performance of this paper, Liu et al.’s scheme[15] and Quijano et al’s scheme[17] as shown in Table 13. Compared with Liu et al.’s scheme[15], the accuracy of this paper is higher 9.66%. Compared with Quijano et al.’s scheme[17], the accuracy of this paper is higher 6.76%.
Performance of basketball hit detection
For the basketball hit prediction model performance of this paper, Liu et al.’s scheme[15] and Quijano et al.’s scheme[17] as shown in Table 14. For basketball hit prediction model, compared with Liu et al.’s scheme[15], the accuracy of this paper is higher 10.72%. Compared with Quijano et al.’s scheme[17], the accuracy of this paper is higher 5.72%.
Performance of basketball hit prediction
Ⅴ. Conclusion
This paper presents a comprehensive computer vision-based basketball player training system. By comparing the accuracy of different methods, the system achieves significant improvements.
For the human skeleton detection model, Utilizing the OpenPose and transfer learning method for human skeleton detection results in a 13% improvement over Liu et al.’s scheme[15] and an 8% improvement over Quijano et al.’s scheme[17].
For the basketball and basketball hoop detection model, using the SSD algorithm and transfer learning for basketball and basketball hoop detection achieves a 7% improvement over Liu et al.’s scheme[15] and an 8% improvement over Quijano et al.’s scheme[17].
For the basketball shot detection model, using mathematical methods results in 9.76% improvements over Liu et al.’s scheme[15] and 7.76% over Quijano et al.’s scheme[17].
For the basketball hit detection model, using mathematical methods results in 9.66% improvements over Liu et al.’s scheme[15] and 6.76% over Quijano et al.’s scheme[17].
For the basketball hit prediction model, using mathematical methods results in 10.72% improvements over Liu et al.’s scheme[15] and 5.72% over Quijano et al.’s scheme[17].
Enhancing the system’s real-time performance and stability to enable efficient operation in practical training scenarios. Exploring more data augmentation techniques and model optimization methods to improve the model’s generalization and robustness. Further advancements and applications of the computer vision-based basketball player training system can be developed and promoted.
References
- B. G. Davis, Tools for Teaching, San Francisco, CA: Jossey-Bass, 1993.
- P. C. Wankat, The Effective, Efficient Professor: Teaching, Scholarship and Service, Boston, MA: Allyn and Bacon, 2002.
-
Z. Liu, “Application of Artificial Intelligence Technology in Basketball Games,” IOP Conference Series: Materials Science and Engineering, Vol. 750, 012093.
[https://doi.org/10.1088/1757-899X/750/1/012093]
-
W. Yan, X. Jiang, and P. Liu, “A Review of Basketball Shooting Analysis Based on Artificial Intelligence,” IEEE Access, Vol. 11, pp. 87344-87365, 2023.
[https://doi.org/10.1109/ACCESS.2023.3304631]
-
H. Meng, “Deep Learning Algorithm and Video Image Processing-Based Basketball Training System,” in Proceedings of 2023 International Conference on Applied Intelligence and Sustainable Computing (ICAISC), Dharwad, India, pp. 1-6, June 2023.
[https://doi.org/10.1109/ICAISC58445.2023.10200635]
-
J. Zhou, “Automatic Detection Approach of Basketball Track in Sports Video,” in Proceedings of 2021 International Conference on Intelligent Transportation, Big Data & Smart City (ICITBS), Xi’an, China, pp. 630-633, March 2021.
[https://doi.org/10.1109/ICITBS53129.2021.00160]
-
R. Ji, “Research on Basketball Shooting Action Based on Image Feature Extraction and Machine Learning,” IEEE Access, Vol. 8, pp. 138743-138751, 2020.
[https://doi.org/10.1109/ACCESS.2020.3012456]
-
W.-L. Tsai, L.-W. Su, T.-Y. Ko, T.-Y. Pan, and M.-C. Hu, “Feasibility Study on Using AI and VR for Decision-Making Training of Basketball Players,” IEEE Transactions on Learning Technologies, Vol. 14, No. 6, pp. 754-762, December 2021.
[https://doi.org/10.1109/TLT.2022.3145093]
-
I. Sassi, S. Anter, and A. Bekkhoucha, “A New Improved Baum-Welch Algorithm for Unsupervised Learning for Continuous-Time HMM Using Spark,” International Journal of Intelligent Engineering and Systems, Vol. 13, No. 1, pp. 214-226, 2020.
[https://doi.org/10.22266/ijies2020.0229.20]
-
E. Epaillard and N. Bouguila, “Variational Bayesian Learning of Generalized Dirichlet-Based Hidden Markov Models Applied to Unusual Events Detection,” IEEE Transactions on Neural Networks and Learning Systems, Vol. 30, No. 4, pp. 1034-1047, April 2019.
[https://doi.org/10.1109/TNNLS.2018.2855699]
-
Z. Li and D. Yu, “Application Prospect of Artificial Intelligence Technology in Vestibular Disorders,” Journal of Clinical Otorhinolaryngology Head and Neck Surgery, Vol. 33, No. 9, pp. 895-897, 2019.
[https://doi.org/10.13201/j.issn.1001-1781.2019.09.025]
-
B. Li and X. Xu, “Application of Artificial Intelligence in Basketball Sport,” Journal of Education, Health and Sport, Vol. 11, No. 7, pp. 54-67, 2021.
[https://doi.org/10.12775/JEHS.2021.11.07.005]
-
P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan, “Object Detection with Discriminatively Trained Part-Based Models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, pp. 1627-1645, September 2010.
[https://doi.org/10.1109/TPAMI.2009.167]
-
Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y. Sheikh, “OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 43, No. 1, pp. 172-186, January 2021.
[https://doi.org/10.1109/TPAMI.2019.2929257]
-
W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “SSD: Single Shot Multibox Detector,” in Proceedings of the 14th European Conference on Computer Vision (ECCV 2016), Amsterdam, Netherlands, pp. 21-37, October 2016.
[https://doi.org/10.1007/978-3-319-46448-0_2]
-
C. Loraksa, S. Mongkolsomlit, N. Nimsuk, M. Uscharapong, and P. Kiatisevi, “Development of the Osteosarcoma Lung Nodules Detection Model Based on SSD-VGG16 and Competency Comparing With Traditional Method,” IEEE Access, Vol. 10, pp. 65496-65506, 2022.
[https://doi.org/10.1109/ACCESS.2022.3183604]
-
W. Liu, K. Quijano, and M. M. Crawford, “YOLOv5-Tassel: Detecting Tassels in RGB UAV Imagery With Improved YOLOv5 Based on Transfer Learning,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 15, pp. 8085-8094, 2022.
[https://doi.org/10.1109/JSTARS.2022.3206399]
저자소개
2017년 7월：Henan University (Bachelor)
2019년 5월：Pace University (Master)
2022년 3월～현 재: 동명대학교 컴퓨터미디어공학과 박사과정
※관심분야：AI, 딥러닝, 빅테이터, CNN
2001년 2월：충북대학교 수학과 (이학박사)
2004년 8월：충북대학교 컴퓨터공학과 (공학박사)
2005년 3월～현 재: 동명대학교 정보보호학과 교수
※관심분야：네트워크 보안, 딥러닝, IoT, 데이터분석