Joseph Williams was born on April 6, 1886, in a small town in Texas called Seguin. Seguin was a nearly desolate town about fifty miles east of San Antonio. Joe Williams' mother, Lettie, was a Native American (reported to be a Cherokee, although never confirmed) and his father was an African American.

In 1905, Williams began his career around the baseball diamonds in San Antonio, Texas. Through four seasons with the San Antonio Black Broncos (1905-1909), the 6' 4", 200 pound pitcher posted records of 28-4, 15-9, 20-8, and 32-8. At one stretch in his career with the San Antonio Black Broncos, he was credited with winning twenty straight games.

In the autumn of 1909, the big, hard throwing pitcher signed to play the winter season with the Trilby's of Los Angeles, California. He remained with the Trilby's, posting outstanding numbers, until the 1910 season when he joined the Chicago Giants. Owner of the Chicago Giants Frank Leland described Joe Williams, by saying "If you have ever witnessed the speed of a pebble in a storm you have not even seen the equal of the speed possessed by this wonderful Texan Giant. He is the king of all pitchers hailing from the Lone Star State and you but to see him once to exclaim, 'That's a plenty!'" Joe Williams received the names "Smokey" and "Cyclone" because of the tremendous amount of speed with which he threw the baseball.

In the winter of 1911, Smokey Joe Williams traveled to Cuba where he compiled a record of 10-7. The following winter, Williams returned to Cuba for another season, this time finishing with a 12-8 record. He sported an overall 22-15 record in Cuban Baseball League action.

In the spring of 1912, Smokey Joe pitched with the Chicago American Giants. He achieved a record of 9-1 this season, which included victories over every Pacific Coast League team except Portland.

After the 1912 season with the Chicago American Giants, Williams left Chicago to join the New York Lincoln Giants where he was with Dick Redding for the first time. Dick Redding, also known as "Cannonball," was one of the fastest throwing pitchers in the history of black baseball. Williams remained with the New York Lincoln Giants through the 1923 season. During his tenure with this team, he either played first base or outfield when he was not pitching. Smokey Joe was also a good hitter with very good power. He posted .269 career batting average with the New York Lincoln Giants. Smokey Joe Williams also served as a manager of the Lincoln Giants for several seasons.

In 1919, after Dick Redding left the New York Lincoln Giants, Williams pitched a no-hitter against Redding's team. The following year, Smokey Joe and Dick Redding feuded and refused to shake hands with each other for a newspaper photographer.

Williams claimed to have pitched five no-hit games in his career, though exact records are not available. His decade of excellence in New York made him a well-known figure in Harlem. Smokey Joe developed the nickname of "Stage Door Johnny" before marrying an ex-showgirl in 1922. In 1924, the New York Lincoln Giants released Williams because they believed, at age 38, his best days were behind him.

In 1925, Smokey Joe Williams signed with the Homestead Grays, a powerful Pittsburgh-based team. With the Grays, Williams experienced an extraordinary late-career resurgence. On August 7, 1930, Smokey Joe pitched perhaps the greatest game in Negro Leagues history. In a night game against the renowned Kansas City Monarchs, Williams allowed only one hit and struck out 27 batters as the Homestead Grays defeated the Monarchs 1-0 in twelve innings. By this time in his career, Joe Williams had become famed as the Methuselah of the Negro Leagues, an image he promoted by claiming to be fifty years old when he was really only forty-four.

In 1932 at the age of forty-six, Williams finally retired from baseball after 28 seasons of pitching. He worked as a bartender in New York where he shared stories with eager customers for many years.

In 1950, when in ill health, Smokey Joe was honored during a Sunday afternoon game at the Polo Grounds. Joseph Williams died on August 25, 1951, in New York. The following year, a Pittsburgh Courier newspaper poll of black baseball officials and sportswriters named Williams the greatest pitcher in the history of the Negro Leagues. Nearly a half century later, on July 25, 1999, Smokey Joe Williams was finally inducted into the Baseball Hall of Fame.