James Potter was born in 1729 in County Tyrone, Ireland. Potter immigrated to Newcastle, Delaware in September 1741, with his father John Potter. In 1746, the family settled down in Cumberland County and in October 1750 Potter's father began serving as High Sheriff. Due to the families constant relocating, Potter did not have a steady education growing up. He first married Elizabeth Cathcart who died young and left behind their two children, Elizabeth and John. Later, Potter married his second wife, widow Mary Patterson, and they had five children: James, Martha, Mary, and Margaret.

When he was 25 years old, Potter served as a Lieutenant in Pennsylvania militia for Northumberland County during the French and Indian War. In 1755, he began serving as captain under General John Armstrong, namesake of Armstrong County, and fought in the victorious 1756 Battle of Kittanning. Armstrong and Potter became close friends after the victory. In 1759, looking down from Nittany Mountain, in an expedition to find an area to build stockade forts, Potter discovered Penns Valley. He was the first to record his travels of Mifflin County and upon looking down at the vast land from Nittany Mountain cried out to his attendant "By heavens, Thompson, I have discovered an empire!" From 1763 to 1764, Potter served on active duty as a major and lieutenant-colonel. He returned to Penns Valley in 1774 to build three stockade forts, one at Old Fort, the next near current Penn Hall, and the third near present day Woodward. In 1777, he also built a stockade fort for travelers called Potter's Fort. Today a "Potter's Fort" historical marker is located roadside near Centre Hall.

At the start of the Revolutionary War, Potter served as a Colonel. He was appointed a brigadier general April 5, 1777, and commanded troops in the counties of Philadelphia, Chester, and Delaware. Potter's service was both distinguished and beneficial as he helped to obtain enemy information for General Washington. As Washington and his army were on the march to Valley Forge on December 11, 1777, Potter and his troops, ahead of Washington, encountered the British enemy proceeding across the Schuylkill River at Matson's Ford (present day West Conshohocken, Montgomery County). Knowing that Washington and his army intended to cross the river, Potter and his troops tried to hold off the British. Potter and his troops persevered but were outnumbered and eventually retreated, destroying their man-made bridge behind them and eventually crossing at Swede's Ford (present day Norristown).

For the Battle at Matson's Ford, as it was later termed, Potter was highly praised by Washington. In The Pennsylvania Magazine of History and Biography author A.B.H. quotes Washington on Potter's attempts to hold off the British, "They were met by General Potter, with part of the Pennsylvania militia, who behaved with great bravery, and gave them every possible opposition until he was obliged to retreat from their superior numbers." After the Battle at Matson's Ford, Potter's wife fell ill and he took leave to take care of her. In the spring of 1778, Washington wrote from Valley Forge, "If the state of General Potter's affairs will admit of his returning to the army, I shall be exceedingly glad to see him, as his activity and vigilance have been much wanted during the winter." Similar letters between Potter and Washington still exist today in the "Papers of George Washington: Revolutionary War Series" at the Constitutional Sources Project.

Potter was very interested in developing the land he held dear in Mifflin County and in 1788 constructed the first house at Potter's Bank. Potter named the log building the Eutaw House after a local Indian tribe and hoped it would serve travelers in Potter County. Potter also constructed saw and grist mills in the area and eventually Potter's Bank became known as Potter Mills. After his death, Potter's grandsons constructed a new Eutaw House as the old one was too small and simplistic. The colonial Eutaw House, located near the original cabin built by Potter, still stands today as a family restaurant. The three Penns Valley forts built by Potter in 1774, however, unfortunately did little to protect settlers from the Indian raids. In 1778, most of Penns Valley's settlers fled Indian attack, including Potter, in an event known as the Great Runaway. In addition to his military involvement, Potter also worked as a farmer throughout his life. General Potter was on the Supreme Executive Council of Pennsylvania in 1780 and in1781 he served as the fourth Vice President of Pennsylvania. In 1782, he was commissioned as major general. In the same year Potter was also elected a candidate for the presidency against John Dickinson. He lost, however, receiving thirty-two votes to Dickinson's forty-one. Potter then became a member of the Council of Censors in 1784, and in 1785 one of the commissioners of rivers and streams.

General Potter did return to his Penn's Valley residence in Mifflin County, now part of Centre County, before his death resulting from injury suffered during a barn-raising. He traveled to Franklin County seeking medical attention and died there in November 1789. He left his Penns Valley 9,000 acre plantation estate to his family, though James Potter Jr., his only son, as the young John Potter had died at age eighteen, inherited most of the land. Potter County was named in General Potter's honor, as well as Centre County's Potters Mills and the General Potter Highway (PA 322 in Centre County).

The namesake of Potter County did survey the land purchased from the Indians in 1768 near to the borders of the county; however, General Potter never actually stepped foot in the area. The original land of Potter County was to be called the Indian name "Sinnemahoning," meaning "Stony Lick," which was the favorable choice of the farmers. A bill of January 13, 1804 proposed the naming of six new counties including Jefferson, McKean, Clearfield, Sinnemahoning, Tioga, and Cambria. However, the state Senate wanted to honor General Potter for his service and on March 26, 1804 the Act was passed and Potter County was named.