American seventh seed Gauff, 18, was among the favourites for the title but was overpowered by an aggressive display from the 17th seed from Latvia.
Rybakina, 23, will play former French Open champion Ostapenko in the quarter-finals instead of the match-up between Swiatek and Gauff that many predicted.
“I am nervous every time I am on court but I try to show calm. It is a big win and I’m happy to be in another round,” Rybakina said.
Poland’s Swiatek rose to the top of the world rankings last year following the retirement of Australia’s Ashleigh Barty, going on to win the French Open and US Open titles as she dominated the WTA Tour.
Rybakina’s success has been more sporadic. Between claiming the most prestigious title in tennis and the start of this year’s Australian Open, she won just 14 of her 24 matches and has spoken out about the lack of recognition she feels she has been given.
She has also struggled with tough draws, with her world ranking lower than expected after receiving no points for her Wimbledon victory.
But she has made another impressive statement at Melbourne Park, reaching the last eight for the first time with a dominant win over Swiatek.
The pair exchanged breaks early in the opening set before a double fault from Swiatek at 3-3 gave Rybakina another chance, which she took with a whopping cross-court backhand winner.
A confident hold to love sealed the one-set advantage, but Swiatek improved at the start of the second set and moved into a 3-0 lead.
Rybakina quickly fought back and levelled for 3-3 as Swiatek struggled to deal with her hefty serve.
She dropped just six points on her first serve in the match and registered an impressive held to love to secure after breaking for 5-4.
“It was a really tough match. I really respect Iga,” said Rybakina.
“Today I think I was serving good. In the important moments I played really well so it made the difference.”