Escaping Non-stationarity

In this image, we study the properties of two artificial intelligence (AI) learning algorithms. The middle image shows the update directions of two algorithms Reg (for regular) and Alt (for alternate). The bottom image shows the variance of the same algorithms. And the bottom-right figure shows the lanscape where these algorithms operate. Our main insight from these figures was that at the corners of the triangles, the updates made by Reg are super small (which effectively leads the algorithm to stall) and the updates for Alt are much bigger (and thus it can make progress). Similarly, at the corners, Regular has small variance and Alternate has big variance—this has similar implications as them having small and large updates. This helped us understand the properies of these two algorithms. In particular, on the problem of MountainCar, we see that Alternate significantly beats Regular (see the top graph), where the Regular, as predicted by the update plots, just stalled. This problem, despite its simplicity, is important as it reflects how Alternate can handle non-stationarity, an important aspect of real world, where the environment, an AI system operates, in continually changes.