American seventh seed Gauff, 18, was among the favourites for the title but was overpowered by the 17th seed from Latvia.
Rybakina, 23, will play former French Open champion Ostapenko in the quarter-finals instead of the match-up between Swiatek and Gauff that many predicted.
“I am nervous every time I am on court but I try to show calm. It is a big win and I’m happy to be in another round,” Rybakina said.
Third seed Jessica Pegula, the highest-ranked player left in the women’s draw, came through 7-5 6-2 against Czech Barbora Krejcikova.
The American will face two-time Melbourne champion Victoria Azarenka in the quarter-finals after Azarenka beat China’s Zhu Lin 4-6 6-1 6-4.
Swiatek ‘wanted it too much’
Poland’s Swiatek rose to the top of the world rankings last year following the retirement of Australia’s Ashleigh Barty, going on to win the French Open and US Open titles as she dominated the WTA Tour.
She reached the Melbourne semi-finals last year and, after a remarkable 2022, returned aiming to be the first women since Serena Williams in 2015 to hold three of the four major titles.
“I felt like I took a step back in terms of how I approach these tournaments and I maybe wanted it a little bit too hard,” she said.
“So I’m going to try to chill out a little bit more.”
Rybakina’s success has been more sporadic. Between claiming the most prestigious title in tennis and the start of this year’s Australian Open, she won just 14 of her 24 matches and has spoken out about the lack of recognition she feels she has been given.
She has also struggled with tough draws, with her world ranking lower than expected after receiving no points for her Wimbledon victory.
But she made another impressive statement at Melbourne Park, reaching the last eight for the first time with a dominant win over Swiatek.
The pair exchanged breaks early in the opening set before a double fault from Swiatek at 3-3 gave Rybakina another chance, which she took with a whopping cross-court backhand winner.
A confident hold to love sealed the one-set advantage, but Swiatek improved at the start of the second set and moved into a 3-0 lead.
Rybakina quickly fought back and levelled for 3-3 as Swiatek struggled to deal with her hefty serve.
She dropped just six points on her first serve in the match and registered an impressive held to love to secure victory after breaking for 5-4.
Rybakina says she is no longer “bothered” about not having earned ranking points at Wimbledon but improving her world status is a source of motivation in Melbourne.
“For sure it’s a motivation, but as I said before, every tournament I come I want to win and no matter points, no points,” she said.
“I love to compete, no matter where I play. For now I would say that I don’t really look at these things.”
Ostapenko has ‘never doubted’ she can win another major
Ostapenko became a teenage champion at French Open almost six years ago but a lack of deep runs since has seen many write her off as a one-hit wonder.
However, against Gauff she displayed the powerful-hitting and aggression that led her to the Roland Garros title in 2017.
Ostapenko hit 30 winners as she broke Gauff’s serve three times and also saved seven of eight break opportunities for the American.
Despite it being the first time since Wimbledon in 2018 that Ostapenko has reached a major quarter-final, the 25-year-old says she “honestly hasn’t really doubted” she could win another.
“My life changed a lot, so I needed a few years to really get used to what happened because I was really young,” said Ostapenko, who had never previously gone beyond the third round at Melbourne Park.
“I always knew and believed in my game. If I play well, I can beat almost anyone.
“I was trying to work more on my consistency, especially in the pre-season, just to step on the court and play my game.”
Gauff had not dropped a set in her previous three matches, including a second-round win over British number one Emma Raducanu.
The teenage prodigy, who lost to Swiatek in last year’s French Open final, became emotional during her news conference as her wait to land a major continues.
“I think every loss is somewhat in my control because I do feel like I’m a good player, but today she just played better,” said Gauff.
“There were moments in the match where I was getting frustrated because I normally can problem-solve, but I feel like I didn’t have much answers to what she was doing.
“There was a couple of things I could have improved on, but overall I think she deserved to win.”