A Few Interesting Facts from the Current World of AI.

A few interesting facts from the current world of AI.
We have a new model that is exceptionally versatile – GATO. This transformer for RL multimodal multi-task reinforcement learning from DeepMind is the only model that can play Atari games, describe images, chat with people, control a real robotic arm, and solve other tasks! This transformer/agent surprises with its versatility.

In April, I wrote about the amazing image generator DALLE-2. Now, its competition comes from Imagen by Google. In fact, there are a few more competitors emerging.

I’m currently taking one of the training courses on transformers from Lazy Programmer (which was released last month), where they start off by glorifying transformers over RNNs, as is customary (yet again referencing the "stolen" paper Attention Is All You Need). RNNs are supposedly much worse than transformers because they lack attention and cannot be computed in parallel. However, an independent researcher, BlinkDL, has emerged, claiming that his RNN combines the best of both RNNs and transformers – excellent performance, fast training, saving VRAM, etc.

An interesting discussion took place on Reddit about how we can trust papers from large laboratories. The author argues that currently, experienced engineers will often just look for ways to squeeze every ounce of performance to make the results in papers look nice, rather than coming up with groundbreaking methods. He demonstrates this with the CIFAR-10 dataset, where they achieved an accuracy of 99.43 (compared to the previous 99.40). They used quite interesting evolutionary algorithms, but the computation of the model took 17,810 TPU core/hours. To give you an idea, this would cost us about 1,350,000 CZK in the cloud, and the result is an improvement of 0.03%.

Sources:
GATO: DeepMind's GATO | Towards AI

Parallelisable RNN: Reddit Discussion

I really don't trust papers from "Top Labs": Reddit Discussion

Attention Is All You Need: arXiv Paper

Originally published on Facebook — link to post