Victoria Manfredi

         Home Research Publications Teaching CV         


NSF Award #2154190

Hierarchical Deep Reinforcement Learning for Routing in Mobile Wireless Networks. This project is a joint collaboration between Wesleyan University and the University of Connecticut.


Synopsis

The use of multi-hop routing in mobile wireless networks is becoming more prevalent, just as these networks are becoming more dense, dynamic, and heterogeneous. Designing a universal multi-hop routing strategy for mobile wireless networks is challenging, however, due to the need to seamlessly adapt routing behavior to spatially diverse and temporally changing network conditions. An alternative to using hand-crafted routing strategies is to use Reinforcement Learning (RL) to learn adaptive multi-hop routing strategies automatically. RL focuses on the design of intelligent agents: an RL agent interacts with its environment to learn a policy, i.e., which actions to take in different environmental states. By using function approximation like deep neural networks (DNNs) as in deep reinforcement learning (DeepRL) to approximate the policy, the RL agent can learn to generalize from its training experience to unseen network conditions and scale the learned routing strategy to larger networks.

The goal of this project is to use DeepRL to develop a universal multi-hop routing strategy for mobile wireless networks that is scalable, generalizable, and adaptive. Specifically, this project will build a novel routing framework that uses hierarchical DeepRL to design an option hierarchy, comprised of multiple layers of routing decisions working together to achieve the overall goals of the network. To enable the same routing strategy to be used at different devices and in unseen network scenarios, the framework will use relational features combined with novel neural network models to handle mobility and perform feature estimation. To further enhance generalizability, the framework will use continual learning to ensure that the routing behaviors learned for more recently seen network scenarios do not dominate the learned routing policy. The developed routing strategies will be thoroughly evaluated using both simulation and experimental testbeds. Through the use of hierarchical DeepRL, this project will provide a significant step forward in developing RL-based routing strategies, and will facilitate development of adaptive strategies for a wide range of mobile wireless networks.


Personnel

Collaborators

Current Students
  • Cheonjin Park, PhD student, University of Connecticut
  • Ishtyaq Mahmud, PhD student, University of Connecticut
  • Zakir Hossein, PhD student, University of Connecticut
  • Brigitte Goehler-Slough, Undergraduate student, Wesleyan University, Graduated 2023 (now Master's student)
  • Haoran Xu, Undergraduate student, Wesleyan University
  • Oliver Diamond, Undergraduate student, Wesleyan University, Graduated 2023 (now resarch assistant)

Past Students
  • Anan Afrida, Undergraduate student, Wesleyan University
  • Jiaxuan Chen, Undergraduate student, Wesleyan University, Graduated 2023
  • Melat Amde Gebremeskel, Undergraduate student, Wesleyan University
  • Jadyn George, Undergraduate student, Wesleyan University
  • Yehrim Hwang, Undergraduate student, Wesleyan University
  • Sydney Keller, Undergraduate student, Wesleyan University

Related Publications
  • Ricci Curvature based Graph Sparsification for Continual Graph Representation Learning
    Xikun Zhang, Dongjin Song, Dacheng Tao
    2023, IEEE Transactions on Neural Networks and Leanring Systems
    [pdf]

  • Multicopy Reinforcement Learning Agents
    Alicia P. Wolfe, Oliver Diamond, Remi Feuerman, Magdalena Kisielinska, Brigitte Goeler-Slough, Victoria Manfredi
    2023, arXiv:2309.10908
    [pdf]

  • Hierarchical Prototype Networks for Continual Graph Representation Learning
    Xikun Zhang, Dongjin Song, Dacheng Tao
    2023, IEEE Transactions on Pattern Analysis and Machine Intelligence. vol. 45, no. 4, pp. 4622-4636, 1 April 2023, doi: 10.1109/TPAMI.2022.3186909.
    [pdf]

  • Sparsified Subgraph Memory for Continual Graph Representation Learning
    Xikun Zhang, Dongjin Song, Dacheng Tao
    2022, IEEE International Conference on Data Mining (ICDM). Location: Orlando, FL. DOI: 10.1109/ICDM54844.2022.00177
    [pdf]

  • Learning an Adaptive Forwarding Strategy for Mobile Wireless Networks: Resource Usage vs. Latency
    Victoria Manfredi, Alicia Wolfe, Xiaolan Zhang, Bing Wang
    2023, arXiv:2207.11386v2, extended version of NeurIPS RL4RealLife workshop paper
    [pdf]

  • Learning an Adaptive Forwarding Strategy for Mobile Wireless Networks: Resource Usage vs. Latency
    Victoria Manfredi, Alicia Wolfe, Xiaolan Zhang, Bing Wang
    2022, Reinforcement Learning for Real Life (RL4RealLife) Workshop in the 36th Conference on Neural Information Processing Systems (NeurIPS)
    [pdf]

  • Relational Deep Reinforcement Learning for Routing in Wireless Networks
    Victoria Manfredi, Alicia Wolfe, Bing Wang, Xiaolan Zhang
    2021, International Symposium on a World of Wireless, Mobile, and Multimedia Networks (WoWMoM)
    [pdf, slides]


Code
  • Under development


Outreach
Mar 2024 Bing Wang described the project in her Advanced Computer Networking course in Spring 2024 at the University of Connecticut.
Feb 2024 Dongjin Song presented a tutorial about continual graph learning at AAAI 2024.
Dec 2023 Dongjin Song gave a talk “Towards Continual Learning on Graphs“ at the University of Arizona, more here.
Dec 2023 Victoria Manfredi gave a talk on “Learning to Route in Mobile Ad Hoc Networks“ at the University of Massachusetts Amherst hosted by Don Towsley.
Nov 2023 Bing Wang described the project in her Computer Networking course in Fall 2023, which was a cross-listed course for both undergraduate and graduate students at the University of Connecticut.
Sep 2023 Victoria Manfredi gave a keynote talk to the IEEE Women in Engineering, IEEE Albuquerque Section, on September 21, 2023, more here.
Aug 2023 Dongjin Song gave a talk “Towards Continual Learning on Graphs“ at the University of Macau, more here.
Aug 2023 Victoria Manfredi was a faculty project leader in the Girls in Science Summer Camp for girls entering 4th-6th grade held at Wesleyan University, more here.
Apr 2023 Dongjin Song introduced the project in his graduate course CSE5713: Data Mining in Spring 2023 at the University of Connecticut.
Apr 2023 Victoria Manfredi described the project in her Computer Networking course in Spring 2023, an undergraduate course at Wesleyan University.
Apr 2023 Victoria Manfredi gave a talk about “Using Machine Learning to Make Decisions in Computer Networks“ at Wesleyan's Natural Sciences and Mathematics Lunch Series on April 14th, 2023.
Feb 2023 Victoria Manfredi gave a talk about “Using Machine Learning to Make Decisions in Computer Networks“ at the CS department of Smith College on February 24th, 2023.
Dec 2022 Dongjin Song has a tutorial about “Continual Graph Learning” accepted to WWW 2023!
Nov 2022 Bing Wang described the project in her Computer Networking course in Fall 2022, which was a cross-listed course for both undergraduate and graduate students.
Nov 2022 Dongjin Song gave an invited about “Continual Graph Learning” at the CS department of University of Central Florida on November 17th, 2022!
Nov 2022 Dongin Song has tutorial about “Continual Graph Learning” accepted to SDM 2023!