Yes Sydney is large place and it depends on where you live.
Most suburbs will have a community pool nearby but some ones of note are
1) Olympic Swim Centre - built for the olympics. Doesn't need much explanation
2) Bondi Icebergs - A hip club where people swim in chilly water all year round
3) North Sydney Olympic Pool - Right below the harbour bridge
4) Cook and Phillip Park Pool - Quite central to the city
On top of that a lot of premium gyms have pools and beaches have wave pools for swimming in the ocean.
There are also plenty of adult sports organisations like https://australia.urbanrec.ca/