Although Isle au Haut is very small, there are several communities here that can be said to have their own identities. Head Harbor is on the southeastern part of the island and, consisting of five or six houses, is a distinct community of it's own. In addition to Head Harbor, there is Moore's Harbor, The Town, The Point, and Rich's Cove (see map).

According to Charles Pratt, if there was ever an area of the island that might be thought of as the "rough section", it is Head Harbor. This reputation, probably long forgotten by most, may have its roots in the "Lobster Wars" of the early 1950s, when some of the fishermen of Penobscot Bay engaged in the sabotage of their competitors equipment. Traps were cut loose, boats, equipment and docks were damaged. It is rumored that certain residents of Head Harbor during that time may have participated in the vandalism. Of course, when one looks out across Head Harbor, it is difficult to think of anything other than the beauty of the place.