advanced
Volume 19 Issue 4
August  2022
Turn off MathJax
Article Contents
Relative Articles
Alessandro Giuseppi, Sabato Manfredi, Antonio Pietrabissa. A Weighted Average Consensus Approach for Decentralized Federated Learning. Machine Intelligence Research, vol. 19, no. 4, pp.319-330, 2022. https://doi.org/10.1007/s11633-022-1338-z
Citation: Alessandro Giuseppi, Sabato Manfredi, Antonio Pietrabissa. A Weighted Average Consensus Approach for Decentralized Federated Learning. Machine Intelligence Research, vol. 19, no. 4, pp.319-330, 2022. https://doi.org/10.1007/s11633-022-1338-z
shu

A Weighted Average Consensus Approach for Decentralized Federated Learning

doi: 10.1007/s11633-022-1338-z
  • 1.

    Department of Computer, Control, and Management Engineering, University of Rome La Sapienza, Rome 00185, Italy

  • 2.

    Department of Electrical Engineering and Information Technology, University of Naples Federico II, Naples 80125, Italy

More Information
  • Author Bio:

    Alessandro Giuseppi received the M. Sc. degree in control engineering and the Ph. D. degree in automatica from University of Rome La Sapienza, Italy in 2016 and 2019, respectively. Since 2016, he has participated in 6 other EU and National research projects, mainly in the fields of control systems and artificial intelligence. He is an assistant professor in automatica at Department of Computer, Control, and Management Engineering Antonio Ruberti (DIAG), University of Rome La Sapienza, Italy. Currently, he is the scientific coordinator of the ESA-funded research project ARIES, related to wildfire emergency management and work package leader in the EU-Korea H2020 project 5G-ALLSTAR. Since 2020, he is serving as associate editor for the International Journal of Control, Automation and Systems (Springer). His main research activities are in the fields of network control and intelligent systems, where he published about 50 papers in international journals and conferences. His research interests include intelligent systems and automatic control, with emphasis on smart networks. E-mail: giuseppi@diag.uniroma1.it (Corresponding author) ORCID iD: 0000-0001-5503-8506

    Sabato Manfredi received the M. Sc. degree in electronics engineering and the Ph. D. degree in computer science and automatica from University of Naples Federico II, Italy in 2001 and 2004, respectively. He is currently an associate professor of automatic control with Department of Electrical Engineering and Information Technology, University of Naples Federico II, Italy. He has been a visiting academic with the Control and Power Group, Electrical and Electronic Engineering Department, Imperial College London, UK since 2012. He has been a visiting professor with School of Mathematical Sciences, Queen Mary, UK, during 2017–2018. He has authored/coauthored more than 90 scientific publications including 18 single-author papers and the monograph: Multilayer Control of Networked Cyber-Physical Systems. Application to Monitoring, Autonomous and Robot Systems (Advances in Industrial Control Series, Springer, 2017). He collaborates with many international universities and companies, holds European patent, is a proponent member of an academic spin-off, and is involved in a range of academic, industrial, and consulting projects. His research interests include automatic control with a special emphasis on nonlinear and complex networks, distributed control and optimization, sensor/drone networks, and new technologies/algorithms for smart city and cyber–physical systems. E-mail: sabato.manfredi@unina.it

    Antonio Pietrabissa received the M. Sc. degree in electronics engineering and the Ph. D. degree in systems engineering from University of Rome “La Sapienza”, Italy in 2000 and 2004, respectively, and where he teaches automatic control and process automation. Since 2000, he has participated in about 25 EU and National research projects. He is associate professor at Department of Computer, Control, and Management Engineering “Antonio Ruberti” (DIAG), University of Rome “La Sapienza”, Italy. Currently, he is the coordinator of the project ARIES on fire emergency prevention, funded by ESA, and the scientific responsible of the research projects 5G-ALLSTARS on 5G communications, funded within the H2020 Europe-South Korea cooperation, and FedMedAI on medical applications of federated learning. He serves as associate editor for Control Engineering Practice (Elsevier) and for IEEE Transactions on Automation Science and Engineering. He is author of more than 50 journal papers and 80 conference papers. His research interest is the application of systems and control theory to the analysis and control of networks. E-mail: pietrabissa@diag.uniroma1.it

  • Received Date: 2022-02-23
  • Accepted Date: 2022-05-05
  • Publish Date: 2022-08-01
    Abstract
  • Federated learning (FedL) is a machine learning (ML) technique utilized to train deep neural networks (DeepNNs) in a distributed way without the need to share data among the federated training clients. FedL was proposed for edge computing and Internet of things (IoT) tasks in which a centralized server was responsible for coordinating and governing the training process. To remove the design limitation implied by the centralized entity, this work proposes two different solutions to decentralize existing FedL algorithms, enabling the application of FedL on networks with arbitrary communication topologies, and thus extending the domain of application of FedL to more complex scenarios and new tasks. Of the two proposed algorithms, one, called FedLCon, is developed based on results from discrete-time weighted average consensus theory and is able to reconstruct the performances of the standard centralized FedL solutions, as also shown by the reported validation tests.

     

    • 1 Actually, in practice, the local weight update is performed iteratively over $ E $ training epochs using a variation of gradient descent (mini-batch gradient descent) that splits $ D_i $ into a set of mini-batches. Equation (3) exemplifies the update rule with E = 1 and over the complete dataset, whereas the pseudo-codes report the mini-batch multi-epoch version of the algorithms.
    FullText(HTML)
  • loading
    • References(49)
  • [1]
    H. B. McMahan, E. Moore, D. Ramage, S. Hampson. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, USA, pp. 1273–1282, 2016.
    [2]
    T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, V. Smith. Federated optimization in heterogeneous networks. In Proceedings of the 3rd Conference on Machine learning and System, Austin, USA, pp. 429–450, 2018.
    [3]
    H. B. McMahan, E. Moore, D. Ramage, B. A. Y. Arcas. Federated learning of deep networks using model averaging. [Online], Available: https://arxiv.org/abs/1602.05629, 2016.
    [4]
    Y. F. Ye, S. Li, F. Liu, Y. H. Tang, W. T. Hu. EdgeFed: Optimized federated learning based on edge computing. IEEE Access, vol. 8, pp. 209191–209198, 2020. DOI: 10.1109/access.2020.3038287.
    [5]
    L. U. Khan, M. Alsenwi, I. Yaqoob, M. Imran, Z. Han, C. S. Hong. Resource optimized federated learning-enabled cognitive internet of things for smart industries. IEEE Access, vol. 8, pp. 168854–168864, 2020. DOI: 10.1109/access.2020.3023940.
    [6]
    A. Hard, K. Rao, R. Mathews, S. Ramaswamy, F. Beaufays, S. Augenstein, H. Eichner, C. Kiddon, D. Ramage. Federated learning for mobile keyboard prediction. [Online], Available: https://arxiv.org/abs/1811.03604, 2018.
    [7]
    T. Yang, G. Andrew, H. Eichner, H. C. Sun, W. Li, N. Kong, D. Ramage, F. Beaufays. Applied federated learning: Improving Google keyboard query suggestions. [Online], Available: https://arxiv.org/abs/1812.02903, 2018.
    [8]
    S. Ramaswamy, R. Mathews, K. Rao, F. Beaufays. Federated learning for Emoji prediction in a mobile keyboard. [Online], Available: https://arxiv.org/abs/1906.04329, 2019.
    [9]
    J. H. Luo, X. Y. Wu, Y. Luo, A. B. Huang, Y. F. Huang, Y. Liu, Q. Yang. Real-world image datasets for federated learning. [Online], Available: https:/arxiv.org/abs/1910.11089, 2019.
    [10]
    L. Ahmed, K. Ahmad, N. Said, B. Qolomany, J. Qadir, A. Al-Fuqaha. Active learning based federated learning for waste and natural disaster image classification. IEEE Access, vol. 8, pp. 208518–208531, 2020. DOI: 10.1109/access.2020.3038676.
    [11]
    T. S. Brisimi, R. D. Chen, T. Mela, A. Olshevsky, I. C. Paschalidis, W. Shi. Federated learning of predictive models from federated electronic health records. International Journal of Medical Informatics, vol. 112, pp. 59–67, 2018. DOI: 10.1016/j.ijmedinf.2018.01.007.
    [12]
    M. J. Sheller, G. A. Reina, B. Edwards, J. Martin, S. Bakas. Multi-institutional deep learning modeling without sharing patient data: A feasibility study on brain tumor segmentation. In Proceedings of the 4th International Workshop on Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, Springer, Granada, Spain, pp. 92–104, 2019. DOI: 10.1007/978-3-030-11723-8_9.
    [13]
    M. Aledhari, R. Razzak, R. M. Parizi, F. Saeed. Federated learning: A survey on enabling technologies, protocols, and applications. IEEE Access, vol. 8, pp. 140699–140725, 2020. DOI: 10.1109/access.2020.3013541.
    [14]
    H. L. Yang, J. Zhao, Z. H. Xiong, K. Y. Lam, S. M. Sun, L. Xiao. Privacy-preserving federated learning for UAV-enabled networks: Learning-based joint scheduling and resource management. IEEE Journal on Selected Areas in Communications, vol. 39, no. 10, pp. 3144–3159, 2021. DOI: 10.1109/JSAC.2021.3088655.
    [15]
    M. Hao, H. W. Li, X. Z. Luo, G. W. Xu, H. M. Yang, S. Liu. Efficient and privacy-enhanced federated learning for industrial artificial intelligence. IEEE Transactions on Industrial Informatics, vol. 16, no. 10, pp. 6532–6542, 2020. DOI: 10.1109/TII.2019.2945367.
    [16]
    K. Wei, J. Li, C. Ma, M. Ding, S. Wei, F. Wu, G. H. Chen, T. Ranbaduge. Vertical federated learning: Challenges, methodologies and experiments. [Online], Available: https://arxiv.org/abs/2202.04309, 2022.
    [17]
    Q. B. Li, Z. Y. Wen, Z. M. Wu, S. X. Hu, N. B. Wang, Y. Li, X. Liu, B. S. He. A survey on federated learning systems: Vision, hype and reality for data privacy and protection. IEEE Transactions on Knowledge and Data Engineering, to be published. DOI: 10.1109/TKDE.2021.3124599.
    [18]
    W. Y. B. Lim, N. C. Luong, D. T. Hoang, Y. T. Jiao, Y. C. Liang, Q. Yang, D. Niyato, C. Y. Miao. Federated learning in mobile edge networks: A comprehensive survey. IEEE Communications Surveys &Tutorials, vol. 22, no. 3, pp. 2031–2063, 2020. DOI: 10.1109/COMST.2020.2986024.
    [19]
    K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, K. Seth. Practical secure aggregation for privacy-preserving machine learning. In Proceedings of ACM SIGSAC Conference on Computer and Communications Security, ACM, Dallas, USA, pp. 1175–1191, 2017. DOI: 10.1145/3133956.3133982.
    [20]
    R. C. Geyer, T. Klein, M. Nabi. Differentially private federated learning: A client level perspective. [Online], Available: https://arxiv.org/abs/1712.07557, 2017.
    [21]
    S. Truex, N. Baracaldo, A. Anwar, T. Steinke, H. Ludwig, R. Zhang, Y. Zhou. A hybrid approach to privacy-preserving federated learning. In Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security, ACM, London, USA, pp. 1–11, 2019. DOI: 10.1145/3338501.3357370.
    [22]
    Q. S. Zhang, B. Gu, C. Deng, H. Huang. Secure bilevel asynchronous vertical federated learning with backward updating. In Proceedings of AAAI Conference on Artificial Intelligence, vol. 35, no. 12, pp. 10896–10904, 2021.
    [23]
    Z. B. Wang, M. K. Song, Z. F. Zhang, Y. Song, Q. Wang, H. R. Qi. Beyond inferring class representatives: User-level privacy leakage from federated learning. In Proceedings of IEEE Conference on Computer Communications, IEEE, Paris, France, pp. 2512–2520, 2019. DOI: 10.1109/infocom.2019.8737416.
    [24]
    S. Kim. Incentive design and differential privacy based federated learning: A mechanism design perspective. IEEE Access, vol. 8, pp. 187317–187325, 2020. DOI: 10.1109/access.2020.3030888.
    [25]
    B. Gu, A. Xu, Z. Y. Huo, C. Deng, H. Huang. Privacy-preserving asynchronous federated learning algorithms for multi-party vertically collaborative learning. [Online], Available: https://arxiv.org/abs/2008.06233, 2020.
    [26]
    J. Konečný, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, D. Bacon. Federated learning: Strategies for improving communication efficiency. [Online], Available: https:/arxiv.org/abs/1610.05492, 2016.
    [27]
    F. Sattler, S. Wiedemann, K. R. Müller, W. Samek. Robust and communication-efficient federated learning from Non-i.i.d. data. IEEE Transactions on Neural Networks and Learning Systems, vol. 31, no. 9, pp. 3400–3413, 2020. DOI: 10.1109/tnnls.2019.2944481.
    [28]
    Q. S. Zhang, B. Gu, C. Deng, S. X. Gu, L. F. Bo, J. Pei, H. Huang. AsySQN: Faster vertical federated learning algorithms with better computation resource utilization. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, ACM, Washington DC, USA, pp. 3917–3927, 2021. DOI: 10.1145/3447548.3467169.
    [29]
    Q. S. Zhang, B. Gu, Z. Y. Dang, C. Deng, H. Huang. Desirable companion for vertical federated learning: New Zeroth-order gradient based algorithm. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, ACM, Atlanta, USA, pp. 2598–2607, 2021. DOI: 10.1145/3459637.3482249.
    [30]
    C. H. Hu, J. Y. Jiang, Z. Wang. Decentralized federated learning: A segmented gossip approach. [Online], Available: https://arxiv.org/abs/1908.07782, 2019.
    [31]
    J. Y. Jiang, L. Hu. Decentralised federated learning with adaptive partial gradient aggregation. CAAI Transactions on Intelligence Technology, vol. 5, no. 3, pp. 230–236, 2020. DOI: 10.1049/trit.2020.0082.
    [32]
    X. H. Chen, J. L. Ji, C. Q. Luo, W. X. Liao, P. Li. When machine learning meets blockchain: A decentralized, privacy-preserving and secure design. In Proceedings of IEEE International Conference on Big Data, IEEE, Seattle, USA, pp. 1178–1187, 2018. DOI: 10.1109/BigData.2018.8622598.
    [33]
    Y. Z. Li, C. Chen, N. Liu, H. W. Huang, Z. B. Zheng, Q. Yan. A blockchain-based decentralized federated learning framework with committee consensus. IEEE Network, vol. 35, no. 1, pp. 234–241, 2021. DOI: 10.1109/MNET.011.2000263.
    [34]
    A. Nedic. Distributed gradient methods for convex machine learning problems in networks: Distributed optimization. IEEE Signal Processing Magazine, vol. 37, no. 3, pp. 92–101, 2020. DOI: 10.1109/MSP.2020.2975210.
    [35]
    S. Manfredi, D. Angeli. Robust distributed estimation of the maximum of a field. IEEE Transactions on Control of Network Systems, vol. 7, no. 1, pp. 372–383, 2020. DOI: 10.1109/TCNS.2019.2906865.
    [36]
    B. D. O. Anderson, M. B. Ye. Recent advances in the modelling and analysis of opinion dynamics on influence networks. International Journal of Automation and Computing, vol. 16, no. 2, pp. 129–149, 2019. DOI: 10.1007/s11633-019-1169-8.
    [37]
    W. Ren. Consensus based formation control strategies for multi-vehicle systems. In Proceedings of American Control Conference, IEEE, Minneapolis, USA, pp. 4237–4242, 2006. DOI: 10.1109/ACC.2006.1657384.
    [38]
    Z. A. Zhang, M. Y. Chow. Convergence analysis of the incremental cost consensus algorithm under different communication network topologies in a smart grid. IEEE Transactions on Power Systems, vol. 27, no. 4, pp. 1761–1768, 2012. DOI: 10.1109/TPWRS.2012.2188912.
    [39]
    C. T. Dinh, N. H. Tran, M. N. H. Nguyen, C. S. Hong, W. Bao, A. Y. Zomaya, V. Gramoli. Federated learning over wireless networks: Convergence analysis and resource allocation. IEEE/ACM Transactions on Networking, vol. 29, no. 1, pp. 398–409, 2021. DOI: 10.1109/tnet.2020.3035770.
    [40]
    R. Olfati-Saber, R. M. Murray. Consensus problems in networks of agents with switching topology and time-delays. IEEE Transactions on Automatic Control, vol. 49, no. 9, pp. 1520–1533, 2004. DOI: 10.1109/TAC.2004.834113.
    [41]
    F. Pedroche, M. Rebollo, C. Carrascosa, A. Palomares. Convergence of weighted-average consensus for undirected graphs. International Journal of Complex Systems in Science, vol. 4, no. 1, pp. 13–16, 2014.
    [42]
    K. Ogata. Discrete-time Control Systems, 2nd ed., New York, USA: Prentice-Hall, Inc., 1995.
    [43]
    F. Haddadpour, M. M. Kamani, A. Mokhtari, M. Mahdavi. Federated learning with compression: Unified analysis and sharp guarantees. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics, Beijing, China, pp. 2350–2358, 2021.
    [44]
    A. Albasyoni, M. Safaryan, L. Condat, P. Richtárik. Optimal gradient compression for distributed and federated learning. [Online], Available: https://arxiv.org/abs/2010.03246, 2020.
    [45]
    L. P. Wang, W. Wang, B. Li. CMFL: Mitigating communication overhead for federated learning. In Proceedings of the 39th IEEE International Conference on Distributed Computing Systems, IEEE, Dallas, USA, pp. 954–964, 2019. DOI: 10.1109/icdcs.2019.00099.
    [46]
    Y. LeCun, C. Cortes, C. J. C. Burges. MNIST handwritten digit database. ATT Labs. [Online], Available: http://yann.lecun.com/exdb/mnist, 2010.
    [47]
    A. Krizhevsky. Learning Multiple Layers of Features from Tiny Images, Technical Report TR-2009, University of Toronto, Toronto, Canada, 2009.
    [48]
    M. Shaha, M. Pawar. Transfer learning for image classification. In Proceedings of the 2nd International Conference on Electronics, Communication and Aerospace Technology, IEEE, Coimbatore, India, pp. 656–660, 2018. DOI: 10.1109/ICECA.2018.8474802.
    [49]
    M. Huh, P. Agrawal, A. A. Efros. What makes ImageNet good for transfer learning? [Online], Available: https://arxiv.org/abs/1608.08614, 2016.
    • Relative Articles
    • Supplements(0)
    • Cited By
  • 加载中

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(8)  / Tables(1)

    Get Citation
    PDF
    XML
    Entire Issue PDF

    用微信扫码二维码

    分享至好友和朋友圈

    Article Metrics

    Article views (939) PDF downloads(79) Cited by()
    Proportional views
    Related
    Browse MIR
    Published Online Current Issue Special Issue Archive Selected Papers
    About MIR
    Aims and Scopes Index information Editorial Board Contact Us Springer Page

    For Authors
    Guide for Authors Submission Template Copyright Agreement
    For Referees
    Guide for Referees Referees Acknowledgement

    MIR Event
    MIR News MIR Awards
    WeChat: 机器智能研究MIR
    Twitter: MIR_Journal

    © Institute of Automation, Chinese Academy of Sciences. Published by Springer Nature and Science Press. All rights reserved. 京ICP备14019135号-25

    Supported by Beijing Renhe Information Technology Co. Ltd  support: info@rhhz.net  

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return