Satellite Integration into 5G: Deep Reinforcement Learning for Network Selection

Emanuele De Santis; Alessandro Giuseppi; Antonio Pietrabissa; Michael Capponi; Francesco Delli Priscoli

doi:10.1007/s11633-022-1326-3

Volume 19 Issue 2

April 2022

Turn off MathJax

Article Contents

Relative Articles

Emanuele De Santis, Alessandro Giuseppi, Antonio Pietrabissa, Michael Capponi, Francesco Delli Priscoli. Satellite Integration into 5G: Deep Reinforcement Learning for Network Selection. Machine Intelligence Research, vol. 19, no. 2, pp.127-137, 2022. https://doi.org/10.1007/s11633-022-1326-3

Citation:

Emanuele De Santis, Alessandro Giuseppi, Antonio Pietrabissa, Michael Capponi, Francesco Delli Priscoli. Satellite Integration into 5G: Deep Reinforcement Learning for Network Selection. Machine Intelligence Research, vol. 19, no. 2, pp.127-137, 2022. https://doi.org/10.1007/s11633-022-1326-3

Citation:

Emanuele De Santis, Alessandro Giuseppi, Antonio Pietrabissa, Michael Capponi, Francesco Delli Priscoli. Satellite Integration into 5G: Deep Reinforcement Learning for Network Selection. Machine Intelligence Research, vol. 19, no. 2, pp.127-137, 2022. https://doi.org/10.1007/s11633-022-1326-3

PDF( 1569 KB)

Satellite Integration into 5G: Deep Reinforcement Learning for Network Selection

doi: 10.1007/s11633-022-1326-3

Department of Computer, Control and Management Engineering “Antonio Ruberti”, University of Rome La Sapienza, Rome 00185, Italy

More Information

Author Bio:
Emanuele De Santis received the B. Sc. degree in automatic control and M. Sc. degree in engineering in computer science with specialization in communication networks control from Sapienza University of Rome, Italy in 2017 and 2019, where he is currently a Ph. D. degree candidate in automatic control. He participated in the H2020 projects 5G-ALLSTAR and 5G-Solutions and in the European Space Agency (ESA) project ARIES. He is a student member of IEEE. His research interests include power and communication network control, artificial intelligence, and optimal control. E-mail: edesantis@diag.uniroma1.it (Corresponding author) ORCID iD: 0000-0003-1011-9737

Alessandro Giuseppi received the B. Sc. degree in computer and automation engineering, the M. Sc. degree in control engineering and the Ph. D. degree in automatica from University of Rome La Sapienza, Italy in 2014, 2016 and 2019, respectively, where he is currently a postdoctoral researcher in automatic control. Since 2016, he has participated in five European and national research projects. He is a member of IEEE. His research interests include network control and intelligent systems. E-mail: giuseppi@diag.uniroma1.it ORCID iD: 0000-0001-5503-8506

Antonio Pietrabissa received the M. Sc. degree in electronics engineering and the Ph. D. degree in systems engineering from Sapienza University of Rome, Italy in 2000 and 2004, respectively. He is an associate professor at Sapienza University of Rome, Italy. He has participated in about 20 European and national research projects. He is a senior member of IEEE. His research interests include the application of systems and control theory to the analysis and control of networks. E-mail: pietrabissa@diag.uniroma1.it ORCID iD: 0000-0003-0188-3346

Michael Capponi received the M. Sc. degree in communication networks control from Sapienza University of Rome, Italy in 2020. Now, he works in a company in the field of computer science.His research interests include reinforcement learning applications to communication networks. E-mail: michaelcapponi96@gmail.com ORCID iD: 0000-0002-5610-9422

Francesco Delli Priscoli received the M. Sc. degree in electronics engineering and the Ph. D. degree in systems engineering from University of Rome, Italy in 1986 and 1991, respectively. From 1986 to 1991, he was with Telespazio, Italy. Since 1991, he has been with University of Rome, Italy, where, at present, he is a full professor of automatic control, control of autonomous multiagent systems, and control of communication and energy networks. He is an Associate Editor of Control Engineering Practice and a Member of the IFAC Technical Committee on Networked Systems. He was/is the Scientific Responsible with University of Rome, for 40 projects funded by the European Union and by the European Space Agency. He is a member of IEEE. His research interests include closed-loop multiagent learning techniques in advanced communication and energy networks. E-mail: dellipriscoli@diag.uniroma1.it ORCID iD: 0000-0001-6140-3661
Received Date: 2021-11-22
Accepted Date: 2022-03-01
Publish Date: 2022-04-01

Abstract

Abstract

This paper proposes a deep-Q-network (DQN) controller for network selection and adaptive resource allocation in heterogeneous networks, developed on the ground of a Markov decision process (MDP) model of the problem. Network selection is an enabling technology for multi-connectivity, one of the core functionalities of 5G. For this reason, the present work considers a realistic network model that takes into account path-loss models and intra-RAT (radio access technology) interference. Numerical simulations validate the proposed approach and show the improvements achieved in terms of connection acceptance, resource allocation, and load balancing. In particular, the DQN algorithm has been tested against classic reinforcement learning one and other baseline approaches.
- Network selection,
- HetNet,
- deep reinforcement learning,
- deep-Q-network (DQN),
- 5G communications

FullText(HTML)

References(33)

References

[1]	3GPP TR 38.811 Study on New Radio (NR) to Support Non-terrestrial Networks, Technical Report. 3GPP, France 2017.
[2]	V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, M. Riedmiller. Playing Atari with deep reinforcement learning, [Online], Available: https://arxiv.org/abs/1312.5602v1, 2013.
[3]	K. S. S. Anupama, S. S. Gowri, B. P. Rao. A comparative study of outranking MADM algorithms in network selection. In Proceedings of the 2nd International Conference on Computing Methodologies and Communication, IEEE, Erode, India, pp. 904−907, 2018. DOI: 10.1109/ICCMC.2018.8487931.
[4]	Y. F. Zhong, H. Q. Wang, H. W. Lv. A cognitive wireless networks access selection algorithm based on MADM. Ad Hoc Networks, vol. 109, Article number 102286, 2020. DOI: 10.1016/j.adhoc.2020.102286.
[5]	S. Radouche, C. Leghris, A. Adib. MADM methods based on utility function and reputation for access network selection in a multi-access mobile network environment. In Proceedings of International Conference on Wireless Networks and Mobile Communications, IEEE, Rabat, Morocco, 2017. DOI: 10.1109/WINCOM.2017.8238177.
[6]	Q. Y. Song, A. Jamalipour. Network selection in an integrated wireless LAN and UMTS environment using mathematical modeling and computing techniques. IEEE Wireless Communications, vol. 12, no. 3, pp. 42–48, 2005. DOI: 10.1109/mwc.2005.1452853.
[7]	T. Ding, L. Liang, M. Yang, H. Q. Wu. Multiple attribute decision making based on cross-evaluation with uncertain decision parameters. Mathematical Problems in Engineering, vol. 2016, Article number 4313247, 2016. DOI: 10.1155/2016/4313247.
[8]	R. K. Goyal, S. Kaushal, A. K. Sangaiah. The utility based non-linear fuzzy AHP optimization model for network selection in heterogeneous wireless networks. Applied Soft Computing, vol. 67, pp. 800–811, 2018. DOI: 10.1016/j.asoc.2017.05.026.
[9]	X. Y. Yan, P. Dong, T. Zheng, H. K. Zhang. Fuzzy and utility based network selection for heterogeneous networks in high-speed railway. Wireless Communications and Mobile Computing, vol. 2017, Article number 4967438, 2017. DOI: 10.1155/2017/4967438.
[10]	M. M. R. Mou, M. Z. Chowdhury. Service aware fuzzy logic based handover decision in heterogeneous wireless networks. In Proceedings of International Conference on Electrical, Computer and Communication Engineering, IEEE, Cox′s Bazar, Bangladesh, pp. 686-691, 2017. DOI: 10.1109/ECACE.2017.7912992.
[11]	A. Wilson, A. Lenaghan, R. Malyan. Optimising wireless access network selection to maintain QoS in heterogeneous wireless environments. In Proceedings of International Symposium on Wireless Personal Multimedia Communications, Aalborg, Denmark. 1236−1240, 2005.
[12]	R. Trestian, O. Ormond, G. M. Muntean. Game theory-based network selection: Solutions and challenges. IEEE Communications Surveys &Tutorials, vol. 14, no. 4, pp. 1212–1231, 2012. DOI: 10.1109/surv.2012.010912.00081.
[13]	J. Antoniou, A. Pitsillides. 4G converged environment: Modeling network selection as a game. In Proceedings of the 16th IST Mobile and Wireless Communications Summit, IEEE, Budapest, Hungary, 2007. DOI: 10.1109/ISTMWC.2007.4299242.
[14]	T. Rahman, M. Z. Chowdhury, Y. M. Jang. Radio access network selection mechanism based on hierarchical modelling and game theory. In Proceedings of International Conference on Information and Communication Technology Convergence, IEEE, Jeju, Korea , pp. 126−131, 2016. DOI: 10.1109/ICTC.2016.7763451.
[15]	L. Rajesh, K. B. Bagan, B. Ramesh. User demand wireless network selection using game theory. In Proceedings of International Conference on Nano-electronics, Circuits & Communication Systems, Jharkhand, India, pp.39−53, 2017. DOI: 10.1007/978-981-10-2999-8_4.
[16]	Meenakshi, N. P. Singh. A comparative study of cooperative and non-cooperative game theory in network selection. In Proceedings of International Conference on Computational Techniques in Information and Communication Technologies, IEEE, New Delhi, India, pp. 612−617, 2016. DOI: 10.1109/ICCTICT.2016.7514652.
[17]	R. S. Sutton, A. G. Barto, Reinforcement Learning: An Introduction, Cambridge, UK: MIT Press, 1998.
[18]	Z. H. Zhang, X. F. Jiang, H. S. Xi. Optimal content placement and request dispatching for cloud-based video distribution services. International Journal of Automation and Computing, vol. 13, no. 6, pp. 529–540, 2016. DOI: 10.1007/s11633-016-1025-z.
[19]	F. S. Lin, B. Q. Yin, J. Huang, X. M. Wu. Admission control with elastic QoS for video on demand systems. International Journal of Automation and Computing, vol. 9, no. 5, pp. 467–473, 2012. DOI: 10.1007/s11633-012-0668-7.
[20]	Z. Y. Du, C. X. Wang, Y. M. Sun, G. F. Wu. Context-aware indoor VLC/RF heterogeneous network selection: Reinforcement learning with knowledge transfer. IEEE Access, vol. 6, pp. 33275–33284, 2018. DOI: 10.1109/access.2018.2844882.
[21]	Y. Yang, Y. Wang, K. Y. Liu, N. Zhang, S. S. Gu, Q. Y. Zhang. Deep reinforcement learning based online network selection in CRNs with multiple primary networks. IEEE Transactions on Industrial Informatics, vol. 16, no. 12, pp. 7691–7699, 2020. DOI: 10.1109/tii.2020.2971735.
[22]	D. D. Nguyen, H. X. Nguyen, L. B. White. Reinforcement learning with network-assisted feedback for heterogeneous RAT selection. IEEE Transactions on Wireless Communications, vol. 16, no. 9, pp. 6062–6076, 2017. DOI: 10.1109/twc.2017.2718526.
[23]	F. Liberati, A. Giuseppi, A. Pietrabissa, V. Suraci, A. Di Giorgio, M. Trubian, D. Dietrich, P. Papadimitriou, F. Delli Priscoli. Stochastic and exact methods for service mapping in virtualized network infrastructures. International Journal of Network Management, vol. 27, no. 6, Article number e1985, 2017. DOI: 10.1002/nem.1985.
[24]	X. W. Wang, J. D. Li, L. X. Wang, C. G. Yang, Z. Han. Intelligent user-centric network selection: A model-driven reinforcement learning framework. IEEE Access, vol. 7, pp. 21645–21661, 2019. DOI: 10.1109/access.2019.2898205.
[25]	K. S. Shin, G. H. Hwang, O. Jo. Distributed reinforcement learning scheme for environmentally adaptive IoT network selection. Electronics Letters, vol. 56, no. 9, pp. 462–464, 2020. DOI: 10.1049/el.2019.3891.
[26]	T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, D. Wierstra. Continuous control with deep reinforcement learning. In Proceedings of the 4th International Conference on Learning Representations, San Juan, Puerto Rico, 2016.
[27]	Y. B. Zhou, Z. M. Fadlullah, B. M. Mao, N. Kato. A deep-learning-based radio resource assignment technique for 5G ultra dense networks. IEEE Network, vol. 32, no. 6, pp. 28–34, 2018. DOI: 10.1109/MNET.2018.1800085.
[28]	B. M. Mao, F. X. Tang, Y. Kawamoto, N. Kato. Optimizing computation offloading in satellite-UAV-served 6G IoT: A deep learning approach. IEEE Network, vol. 35, no. 4, pp. 102–108, 2021. DOI: 10.1109/MNET.011.2100097.
[29]	E. De Santis. Trunk96/wireless-network-simulator, [Online], Available: https://github.com/trunk96/wireless-network-simulator, 2022.
[30]	F. D. Priscoli, A. Giuseppi, F. Liberati, A. Pietrabissa. Traffic steering and network selection in 5G networks based on reinforcement learning. In Proceedings of European Control Conference, IEEE, St. Petersburg, Russia, pp. 595−601, 2020. DOI: 10.23919/ECC51009.2020.9143837.
[31]	5G; NR; Physical Channels and Modulation, ETSI TS 138 211 v15.2.0. 3GPP, 2018.
[32]	Final report for COST Action 231, [Online], Available: http://www.lx.it.pt/cost231/final_report.htm, 2022.
[33]	G. Maral, M. Bousquet, Z. L. Sun. Satellite Communications Systems: Systems, Techniques and Technology. 6th ed., Hoboken, USA: Wiley, 2020. DOI: 10.1002/9781119673811.

Relative Articles

Supplements(0)

Cited By

Proportional views

Proportional views

通讯作者: 陈斌, bchen63@163.com

1.
沈阳化工大学材料科学与工程学院沈阳 110142

Figures(12) / Tables(1)

Get Citation

PDF

XML

Entire Issue PDF

用微信扫码二维码

分享至好友和朋友圈

Article Metrics

Article views (308) PDF downloads(37)

Satellite Integration into 5G: Deep Reinforcement Learning for Network Selection

doi: 10.1007/s11633-022-1326-3

Abstract

References

Proportional views

Catalog

通讯作者: 陈斌, bchen63@163.com

Article Metrics

Proportional views

Related

Satellite Integration into 5G: Deep Reinforcement Learning for Network Selection

doi: 10.1007/s11633-022-1326-3

Abstract

References

Proportional views

Catalog

通讯作者: 陈斌, bchen63@163.com

Article Metrics

Proportional views

Related

Export File

Citation

Format

Content