Sillman was born in Savannah, Georgia. He enrolled at Black Mountain College after he served in World War II. His intention was to study architecture but soon found an interest in color, shapes and the use of line.
Sillman was a student of Josef Albers and eventually followed Albers to Yale University where he pursued a BFA (1951) and a MFA (1953). He co-founded Ives/Sillman Publication and printed editions for many artists including Romare Bearden, Jacob Lawrence and Josef Albers.
Sillman was widely exhibited during his lifetime and is represented in the permanent collections of the Museum of Modern Art, The Whitney Museum, and the Phillips Collection.