The brown fur seal is the largest and most robust fur seal. It has a large and broad head with a pointed snout that may be flat or upturned slightly. They have external ear flaps (pinnae) and their whiskers (vibrissae) are long, and may extend backward past the pinnae, especially in adult males. The foreflippers are covered with sparse hair over about three-quarters of their length. The hindflippers are short relative to the large body, with short, fleshy tips on the digits. The size and weight of the brown fur seal depends on the subspecies. The Southern African subspecies is on average slightly larger than the Australian subspecies. Males of the African subspecies (A. p. pusillus) are 2.3 metres (7.5 ft) in length on average and weigh from 200–300 kilograms (440–660 lb). Females are smaller, averaging 1.8 metres (5.9 ft) in length and weighing an average of 120 kilograms (260 lb). Males of the Australian subspecies (A. p. doriferus) are 2–2.2 metres (6.6–7.2 ft) in length and weigh 190–280 kilograms (420–620 lb). Females are 1.2–1.8 metres (3.9–5.9 ft) length and weigh 36–110 kilograms (79–240 lb).
Adult male brown fur seals are dark gray to brown, with a darker mane of short, coarse hairs and a light belly, while adult females are light brown to gray, with a light throat and darker back and belly. The foreflippers of the fur seal are dark brown to black. Pups are born black and molt to gray with a pale throat within three to five months. The skull of the African subspecies has a larger crest between the mastoid process and the jugular process of the exoccipital.