{"id":1938,"date":"2025-09-04T08:31:40","date_gmt":"2025-09-04T07:31:40","guid":{"rendered":"https:\/\/babel.isa.uma.es\/kipr\/?p=1938"},"modified":"2025-09-04T08:31:40","modified_gmt":"2025-09-04T07:31:40","slug":"on-the-theoretical-convergence-of-q-learning-when-the-environment-is-not-stationary","status":"publish","type":"post","link":"https:\/\/babel.isa.uma.es\/kipr\/?p=1938","title":{"rendered":"On the theoretical convergence of Q-learning when the environment is not stationary"},"content":{"rendered":"\n<h4 class=\"wp-block-heading\">Diogo S. Carvalho, Pedro A. Santos, Francisco S. Melo, <strong>Reinforcement learning in convergently non-stationary environments: Feudal hierarchies and learned representations,<\/strong> Artificial Intelligence, Volume 347, 2025, <a href=\"https:\/\/doi.org\/10.1016\/j.artint.2025.104382\">10.1016\/j.artint.2025.104382<\/a>.<\/h4>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">We study the convergence of Q-learning-based methods in convergently non-stationary environments, particularly in the context of hierarchical reinforcement learning and of dynamic features encountered in deep reinforcement learning. We demonstrate that Q-learning achieves convergence in tabular representations when applied to convergently non-stationary dynamics, such as the ones arising in a feudal hierarchical setting. Additionally, we establish convergence for Q-learning-based deep reinforcement learning methods with convergently non-stationary features, such as the ones arising in representation-based settings. Our findings offer theoretical support for the application of Q-learning in these complex scenarios and present methodologies for extending established theoretical results from standard cases to their convergently non-stationary counterparts.\n<\/p>\n<\/blockquote>\n","protected":false},"excerpt":{"rendered":"<p>Diogo S. Carvalho, Pedro A. Santos, Francisco S. Melo, Reinforcement learning in convergently non-stationary environments: Feudal hierarchies and learned representations, <span class=\"ellipsis\">&hellip;<\/span> <span class=\"more-link-wrap\"><a href=\"https:\/\/babel.isa.uma.es\/kipr\/?p=1938\" class=\"more-link\"><span>Read More &rarr;<\/span><\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[44],"tags":[584,13],"class_list":["post-1938","post","type-post","status-publish","format-standard","hentry","category-reinforcement-learning-theory","tag-convergence-of-rl","tag-q-learning"],"_links":{"self":[{"href":"https:\/\/babel.isa.uma.es\/kipr\/index.php?rest_route=\/wp\/v2\/posts\/1938","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/babel.isa.uma.es\/kipr\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/babel.isa.uma.es\/kipr\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/babel.isa.uma.es\/kipr\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/babel.isa.uma.es\/kipr\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1938"}],"version-history":[{"count":1,"href":"https:\/\/babel.isa.uma.es\/kipr\/index.php?rest_route=\/wp\/v2\/posts\/1938\/revisions"}],"predecessor-version":[{"id":1939,"href":"https:\/\/babel.isa.uma.es\/kipr\/index.php?rest_route=\/wp\/v2\/posts\/1938\/revisions\/1939"}],"wp:attachment":[{"href":"https:\/\/babel.isa.uma.es\/kipr\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1938"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/babel.isa.uma.es\/kipr\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1938"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/babel.isa.uma.es\/kipr\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1938"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}