Summary: Advice on Reinforcement Learning Experimentation

During the BeNeRL seminar talks researchers also briefly share their approach to RL experimentation. The below list keeps track of their main advice.
(BE = Benjamin Eysenbach)

Managing experiments

1. Maintain a lab/experiment journal (BE)

• Number every experiment you perform

• Rule of thumb: a paper requires in total ~200-250 experiments (unfruitful research directions will stop earlier)

2. Before each experiment, write down your hypotheses (BE)

• Determine what you want to get out of the next experiment

• Think beyond "does my method outperform the baseline?" --> there are many more (interesting) questions

3. Add reminders to yourself (BE)

• In your journal, add notes to yourself: when to check back to a certain experiment (with a date), what to look for, what to do next, etc. 

Interpreting experiments

1. Log as much data as possible (BE)

• Don't only log learning curves. The more information you log, the more you can analyse

2. Dig deep into your results (BE)

• Analysis > Coding. Spend a significant part of your time on analyzing the output of an experiment

• Be curious, try to learn what is going on. Don't only look at learning curves

3. Visualize (BE)

• Try to visualize your results as much as possible