How Does the AI Salesforce Text Generation Model Work?

Just under a month ago, the AI Salesforce team released a new text generation model called CTRL.

The algorithm works by allowing you to input a web link, and it attempts to generate an article that could be found at that link. This model is truly massive—it is built on 1.6 billion parameters and is trained on three times the amount of data as GPT-2 (which was 8 million of the best pages from Reddit).
I will demonstrate how it works using two links that I invented:

https://techcrunch.com/…/…/best-mobile-game-with-advanced-ai
The Best Mobile Game with Advanced AI
In recent years, we have seen many games attempting to do something new and different. One such example is "Minecraft," which has been around for quite some time but recently added an interesting twist using artificial intelligence (AI) to assist players in building things like structures, weapons, and other items.
However, this is not just any old AI. It is referred to as AI. It is used not only in Minecraft—although that is where most people will first look—but also in many other games, including Angry Birds, Star Wars, and Clash Royale.
What makes this particular type of AI special is its ability to learn from experience.
The team at DeepMind Labs has created a system that can play thousands of levels of Minecraft without getting bored. This is achieved through learning the mechanics of the game over time, as well as analysing data collected during gameplay.
***end***
And here’s another link that I again invented, along with the article generated by the algorithm:

https://www.cnn.com/…/beautiful-naked-girle-found-on-church…
Naked Woman Found on Church Roof
MUST SEE
(CNN) A naked woman was discovered on Sunday morning by a priest and his wife in the tower of St. Peter's Basilica in the Vatican, according to Italian media. The woman, believed to be in her thirties, was seen earlier that day sitting atop the church tower with her hands clasped over her knees, said Monsignor Pietro Parolin. He told CNN that he did not know if she would have any identification or how long it would take for the police to identify her.

Read more

“All we can say right now is that this person likely just came from elsewhere,” Parolin said. The woman's body disappeared before dawn on Sunday, but the police are still searching for her, he added. The police are investigating whether other people were involved in the incident. This occurred while Pope Francis was celebrating Mass in the basilica—one of the holiest sites in Catholicism—after returning from an international trip where he visited several countries, including South Korea, Israel, and Jordan.
***end***

It’s not perfect, but it’s still one of the best AI fake news articles I’ve seen. It maintains context throughout the entire article and generates articles with great sensitivity to the server where it is supposedly located. The algorithm can also be asked questions. It correctly answered who the first Czech president was—Václav Havel. When I asked what the meaning of life is, it replied that it is life. When I asked which metal is more expensive than gold, it answered nickel (here it clearly made a mistake—I checked and it is currently being bought for around 200 CZK per kilo:))

So, what do you think of the new CTRL model?

Sources:
Github: https://github.com/salesforce/ctrl
Collab to try it out: https://colab.research.google.com/…/1hVveBQShDru1Mjnhe4C21u…
Maxík's article: https://minimaxir.com/2019/09/ctrl-fake-news/