From zero to hero: Reinforcement learning insights from the Ai4 conference

Last week at Ai4 in Las Vegas I moderated a panel on reinforcement learning, “Zero to Hero: A Data Scientist’s Introduction to Reinforcement Learning.” The panel was a mix of academics, practitioners, and researchers: William Tran, Senior Engineering Manager at Pager Health, Jasleen Singh, Senior Principal Research Scientist at Dell, Alfredo Garcia, Professor at Texas A&M, and Sainyam Galhotra, Assistant Professor at Cornell. Here, I’ll share the key insights and takeaways from our discussion.

What is reinforcement learning?

Reinforcement learning (RL) is a branch of machine learning where an agent learns to make decisions by interacting with an environment. The agent receives feedback in the form of rewards or penalties and aims to maximize cumulative rewards. This approach is distinct from supervised learning—which relies on labeled data—as RL teaches through the consequences of actions, making it suitable for dynamic and complex environments.

What is reinforcement learning useful for?

RL is particularly useful in scenarios that require complex decision-making and adaptability, such as:

• Gaming: RL has achieved superhuman performance in games like Go and chess by learning strategies through self-play.
• Robotics: RL enables robots to adapt to changing environments, improving their ability to perform tasks autonomously.
• Autonomous vehicles: RL assists in developing systems that can navigate and make decisions in real time.

However, its application in other fields is limited because of ethical concerns and the need for safe experimentation environments. Taking healthcare as an example, RL runs into the following issues:

• Ethical concerns: In healthcare, the stakes are incredibly high, and patient safety is paramount. This means that experimentation, which is a core component of RL, is often considered an unacceptable risk in real-world medical settings. The ethical implications of allowing an RL agent to explore different treatment options without guaranteed safety can be significant.
• Lack of simulated environments: Unlike gaming or robotics, where environments can be easily simulated, healthcare lacks comprehensive simulations that can accurately replicate real-world scenarios. This makes it challenging to train RL models safely and effectively without risking patient safety.
• Data limitations: Healthcare data is often incomplete or unavailable because of privacy concerns, making it difficult to train RL models. Additionally, the dynamic nature of medical environments means that data from past cases may not always apply to new situations.

Despite these challenges, there are promising areas where RL can be applied in healthcare:

• Drug discovery: RL can be used to explore new drug combinations and optimize treatment protocols by simulating various scenarios and learning from outcomes. This application requires collaboration with domain experts to define reward functions and simulate environments effectively.
• Personalized medicine: RL has the potential to tailor treatments to individual patients by learning from their specific responses to therapies. This could lead to more effective and efficient healthcare delivery.

To read more about trends and opportunities in healthcare, check out SignalFire’s perspective here.

‎Getting started with reinforcement learning

For newcomers to RL, several resources can help you get started:

• Online courses: Stanford's CS234 by Emma Brunskill is a highly recommended introduction to RL.
• Open source libraries: Libraries like OpenAI's Gym provide environments to experiment with RL algorithms.
• YouTube tutorials: Numerous tutorials are available to provide a foundational understanding of RL concepts such as ”Introduction to Reinforcement Learning” by David Silver.

Reinforcement learning presents several challenges

For those beginning to learn in this space, be prepared for the following:

• Computational intensity: RL requires significant computational resources because of its iterative nature.
• Simulating environments: Creating realistic simulations is crucial but can be complex and resource-intensive.
• Exploration vs. exploitation: Balancing the exploration of new strategies and the exploitation of known ones is a core challenge in RL.

Reinforcement learning from human feedback (RLHF)

It is a technique used to enhance machine learning models, especially large language models (LLMs) and other AI systems, by incorporating human feedback into the reinforcement learning process.

How is RLHF different from traditional RL

Reinforcement learning from human feedback and traditional reinforcement learning both involve training models to make decisions based on rewards. However, they differ significantly in how rewards are defined and how the training process is guided. 

• Reward signal: In traditional RL, rewards come from the environment and are predefined by experts, reflecting desired outcomes (e.g., game scores). In RLHF, rewards are based on human feedback, used to train a reward model that assesses the quality or desirability of outputs, aligning the model with human values and preferences.
• Training process: In traditional RL, the model learns through trial and error from environment interactions, with rewards based on performance. This can lead to suboptimal behaviors if the reward structure is poorly designed. In RLHF, the model is pre-trained with supervised learning and then fine-tuned with reinforcement learning using human feedback. This feedback creates a reward model that aligns the training with human values and helps refine the model’s behavior.
• Application and use cases: Traditional RL is applied in environments where rewards are explicitly defined, such as game playing, robotics, and optimization tasks, helping models learn optimal strategies through trial and error. RLHF, on the other hand, is employed when aligning a model's outputs with human preferences is critical, such as in conversational agents and systems requiring ethical considerations, using human feedback to refine responses and ensure they meet human values.

The importance of interdisciplinary research

Interdisciplinary collaboration is vital for advancing RL. Insights from neuroscience, for example, can inform the development of models that mimic human learning processes. In domains like drug discovery, collaboration with domain experts is essential to define reward functions and simulate environments effectively.

Control theory intersects with RL, particularly in optimal control problems where the goal is to maneuver systems efficiently, such as operating spacecraft with minimal fuel usage. RL offers solutions for environments with unknown variables, making it valuable for control engineers dealing with "unknown unknowns" in dynamic systems.

Trends to watch in reinforcement learning

The panelists highlighted several exciting trends, including:

• Contextual reinforcement learning: Leveraging existing data to provide context and improve RL efficiency
• Safety in RL: Developing algorithms that prioritize safety, especially in applications like robotics and healthcare
• Reinforcement learning from human feedback (RLHF): Integrating human feedback to iteratively refine models, offering a promising avenue for improving RL systems' performance
• Inverse reinforcement learning (IRL): Another exciting area where the goal is to infer the reward function, given observed behavior—particularly useful when it’s challenging to specify the reward function, as it allows for a more compact representation of the desired behavior

As RL continues to evolve, its potential to transform industries by enabling machines to learn and adapt is immense. Addressing the challenges of exploration, safety, and interdisciplinary collaboration will be key to unlocking this potential.

If you have questions or want to share what you’re working on, feel free to email me at oana@signalfire.com. Thanks to  William, Jasleen, Alfredo, and Sainyam for their input and for their authentic conversation on stage at Ai4.