The novel provides a detailed, episodic record of the two branches of the wealthy and aristocratic Jia (賈) clan — the Rongguo House (榮國府) and the Ningguo House (寧國府) — who reside in two large, adjacent family compounds in the capital Beijing. Their ancestors were made Dukes and given imperial titles, and as the novel begins the two houses are among the most illustrious families in the city. One of the clan’s offspring was made an Imperial Consort, and a lush landscaped garden was built to receive her visit. The novel describes the Jias’ wealth and influence in great naturalistic detail, and charts the Jias’ fall from the height of their prestige, following some thirty main characters and over four hundred minor ones. Eventually the Jia clan falls into disfavor with the Emperor, and their mansions are raided and confiscated.

In the novel's frame story, a sentient Stone, abandoned by the goddess Nüwa when she mended the heavens aeons ago, begs a Taoist priest and a Buddhist monk to bring it with them to see the world. The Stone, accompanied by a character named Divine Attendant-in-Waiting (神瑛侍者) (while in Cheng-Gao versions they are merged into the same character), was given a chance to learn from the human existence, and enters the mortal realm.

The main character of the novel is the carefree adolescent male heir of the family Jia Baoyu. He was born with a magical piece of "jade" in his mouth. In this life he has a special bond with his sickly cousin Lin Daiyu, who shares his love of music and poetry. Baoyu, however, is predestined to marry another cousin, Xue Baochai, whose grace and intelligence exemplifies an ideal woman, but with whom he lacks an emotional connection. The romantic rivalry and friendship among the three characters against the backdrop of the family's declining fortunes forms the main story in the novel.