June 11, 2022·Rosie·2 min read·Archive 2022

A Few Interesting Facts from the Current World of AI

We have a new model that is exceptionally versatile – GATO. This transformer on RL multimodal multi-task reinforcement learning from DeepMind. The only model that…

We have a new model that is exceptionally versatile – GATO. This transformer on RL multimodal multi-task reinforcement learning from DeepMind is the only model that can play Atari games, describe images, chat with people, control a real robotic arm, and tackle various other tasks! This transformer/agent surprises with its versatility.

In April, I wrote about the amazing image generator DALLE-2. Now, its competition is coming from Imagen by Google. In fact, there are a few more competitors emerging.

I am currently taking one of Lazy Programmer's training sessions on transformers (which was released last month), where they typically start by glorifying transformers over RNNs, as is customary everywhere (for the umpteenth time referencing the "stolen" paper Attention Is All You Need). RNNs are allegedly much worse than transformers because they lack attention and cannot be computed in parallel. However, an independent researcher, BlinkDL, has emerged, claiming that his RNN combines the best of both RNNs and transformers – excellent performance, quick training, saving VRAM, and so on.

An interesting discussion took place on Reddit about how we can trust papers from large labs. The author argues that experienced engineers nowadays often just look for ways to squeeze every ounce of performance to make the results in papers look nice, rather than coming up with groundbreaking methods. He demonstrates this with the CIFAR-10 dataset, where they achieved an accuracy of 99.43 (compared to the previous 99.40). They used quite interesting evolutionary algorithms, but the computation of the model took 17,810 TPU core hours. To give you an idea, this would cost us about 1,350,000 CZK in the cloud, and the result is an improvement of just 0.03%.

A Few Interesting Facts from the Current World of AI