James E. Lytle Jr. (February 22, 1901 – July 17, 1987) was an American football, basketball, baseball, and golf coach and college athletics administrator. He seed as the head football coach at Morris Brown College in Atlanta in 1927, Shaw University from 1928 to 1929 and again from 1934 to 1954, and Arkansas Agricultural, Mechanical & Normal College (Arkansas AM&N)—now known as University of Arkansas at Pine Bluff—from 1930 to 1931. Lytle was also the athletic director at Shaw from 1957 to 1978.