Part of the problem has to do with irresponsible fishing practices, such as trawling, which essentially scoops up everything within range from the ocean floor -- only about 5% of the contents are used, while the other 95% is dumped, dead, back into the ocean. Another problem is that fisheries are simply taking too many fish.
Studies show it takes industrial fisheries only 10 to 15 years to diminish any new fish community they encounter to one-tenth of its original size.
Much of the fish caught nowadays (about one-third) are actually not used for human food at all, but are ground up to be used as agricultural or aquacultural feed.