Jimmy Louis Escalle was born on November 7, 1929 in Fresno, California, but grew up in the small farming community of Earlimart, about 60 miles south of Fresno. He lived in a two-room house that he shared with his father, mother, and younger brother, Robert.

He went to grammar school in Earlimart, but attended and graduated from Delano Joint Union High School, which is about 7 miles south of Earlimart.

After graduating from high school in June 1947, he went to Visalia Junior College for a semester before transferring to Bakersfield College, where he graduated in June 1950. This was just two weeks before North Korea invaded South Korea.

Jim majored in mechanical engineering and planned on continuing his education at the University of California at Berkeley. He attended for one semester, but then decided to apply for the Aviation Cadet program with the U.S. Air Force.

He was accepted into the program, but while waiting for an opening in a cadet class, Jim was told his draft number would be coming up in the near future. So he decided to do what others did in his situation. He enlisted in the Air Force as a private and reported to Lackland Air Force Base for basic training.