That has prompted Royal Life Saving Australia to urge enthusiasts to wear life jackets, check weather warnings and not fish if conditions are dangerous.
The latest death was that of a fisherman swept into the water on the NSW far south coast on Saturday morning.
Emergency services were called to Kianinny Bay near Kangarutha Walking Track, Tathra, with the multi-agency operation locating his body in nearby waters about 10.40am.
In Victoria on Friday morning, emergency workers were called to a beach near San Remo, 125km southeast of Melbourne, after reports three people were washed into the water.
One woman made it back to shore while an aerial search found a second unresponsive in the water.
She could not be revived and a search continued on Saturday for a missing man.
Also on Friday morning, a fisherman died after being swept into Wollongong Harbour.
Emergency services were called to Endeavour Drive after 6am following reports a man was pulled from the water unconscious. The 58-year-old could not be revived.
The same morning a man drowned at Mosman on Sydney's north shore.
A body recovered after a fisherman was swept off rocks at Green Cape near Eden on Friday afternoon was yet to be identified and a search meanwhile continued for a man lost in heavy seas off Sydney's Eastern Suburbs.
Rescuers were called to Little Bay Beach about 1pm on Friday after the 24-year-old, walking on rocks with friends, was washed into the water by a large wave and failed to resurface.
A multi-agency search commenced and was resumed on Saturday.
The Bureau of Meteorology cautioned residents to take extra care along major parts of the NSW coast.
"Surf and swell conditions are expected to be hazardous for coastal activities such as rock fishing, boating and swimming," the bureau said in a statement.
Royal Life Saving Australia CEO Justin Scarr said past investigations had prompted strong advice for rock fishers to wear life jackets and safety equipment but most importantly check weather conditions.
"What we've had is significant swell that has meant almost no rock shelf between the Queensland border and Melbourne is safe for rock fishing," he told AAP.
Dr Scarr said people on Sunday and Monday should think twice about rock fishing and check the weather warnings, particularly wave action and direction.
He urged parents to constantly watch children around water and said boaties needed to have enough life jackets aboard.
Over the past 20 years, 118 Australians have drowned during Easter, according to the RLSA.
19°C