(or "If At First You Don't Succeed...")
For those of us who came of age as comics fans in the 80s and 90s, the thought that there was ever a time when the X-Men weren't immensely popular is a bit of a tricky one to get our heads around. They are, after all, the X-Men, the all-conquering comics juggernaut that single-handedly dominated market shares for the better part of a decade. How could there be a time, historically, when the title was a struggling bi-monthly that resorted to reprints for four-and-a-half years and was on the relative verge of cancellation? What was it that made one X-Men series such a hit and another not? What exactly did they add and remove to the X-Men's storytelling engine to turn it from dud to hit? It's going to be impossible to pin it down to a single element, of course, but let's look at some of the potential causes.
A big contender right off the bat is "tone". The style and emotional content of a series comes up only occasionally in these columns, if for no other reason than it's difficult to pin down when compared to the more concrete elements that stand out. But there really shouldn't be any doubt that Stan Lee and Chris Claremont are very different writers, and that the Silver Age and the Bronze Age were very different stylistic eras. Perhaps the central concept of the X-Men, with its emphasis on generational gaps and parallels to racial issues, was just inherently more suited to a Bronze Age writer that emphasized characterization rather than a Silver Age writer who emphasized action. Maybe it just needed time to come into its own.
The rogues gallery also deserves mention; sure, the first 66 issues of X-Men brought us Magneto, the Juggernaut, and the Sentinels, but it also brought us "Grotesk the Sub-Human", "Unus the Untouchable", Lucifer, and a big Frankenstein Monster robot built by aliens. Very few of the X-Men's foes from this era were interesting, credible threats, and of those few, even fewer really felt like they belonged in an X-Men series.
Ultimately, the one I think comes closest to being the root cause (and bearing in mind, this is only a personal opinion)...the team dynamic really falls flat compared to the other Lee/Kirby teams of the era. The Beast is a great character--someone who really comes alive on the page--but Cyclops and Angel seem fairly interchangeable, Iceman doesn't have the charisma that helps keep Johnny Storm from getting irritating, and Marvel Girl has a personality similar to pretty much any other 60s female character in comics. There's nobody to lend the team spark and conflict, like Hawkeye did for the Avengers, or the Thing did for the Fantastic Four, or...like Wolverine would later for the X-Men.
So why, then, did Marvel stick with the title? If you have to bring in a new creative team, jettison four of the five central characters, and redraft the team dynamic from the ground up, why are you still calling the team the X-Men? (Apart, of course, from the name-recognition factor and protection of trademarks.) What was the good thing about the storytelling engine that made them decide to try to salvage it?
Fundamentally, the X-Men just have a very good central concept, one that's interesting and unusual and opens up a lot of storytelling possibilities. A wise old man with amazing powers finds young people who are just developing their own startling abilities and takes them in at his school, both to teach them how to use their powers and to teach them how to protect the world against those who would use these abilities for evil. Someone must have recognized that a concept this good deserves a second chance.