Understanding reinforcement learning
Reinforcement learning and PPO in plain language, and how what a model learns in simulation carries over to the real world.
What reinforcement learning is
Ordinary machine learning needs someone to write down the right answers first. Reinforcement learning has no answer key, only a score: do well and gain points, do badly and lose them. The model has to try things and work out from the score which actions were better. That is why driving explains it so well -- nobody could list the correct steering angle for every situation on a track, but anyone can tell a good lap from a bad one.
What the reward function does
The reward function is your scoring rule, and the only instruction you can really give the model. Score speed highly and it starts cutting corners; penalise touching the line and it learns to trade speed against staying clean. Writing the reward often matters more than tuning parameters, because the model will always head for the highest score -- including through loopholes you did not think of.
What PPO is
PPO (Proximal Policy Optimization) is one of the most widely used reinforcement learning algorithms. Its central idea is a practical one: never change too much at once. Change too little and learning crawls; change too much and a single update can destroy everything learned so far. PPO keeps each step inside a bound, which makes training far steadier than the methods that came before it.
Why simulation and reality differ
This gap has a name: the sim-to-real gap. Floor friction in a simulator is one fixed number; a real floor is different everywhere. A simulated motor turns exactly as fast as it is told; a real one slows as the battery drains. Part of a model's good simulated score comes from genuinely learning to drive, and part comes from learning to exploit the simulator. Putting the car on a real track is the most direct way to tell the two apart.