The first two Harry Potter films overcame major hurdles to become blockbuster hits. Producer David Heyman and director Chris Columbus, working with author J.K. Rowling, created movies that were true to the spirit of the books. At the same time, the debut films began to create their own vision of Rowling’s characters.

That was just the first step, however, in creating a series we’ll be watching for decades to come. When Columbus decided not to return for a third movie, the producers had the chance to be daring. Their choice to direct Harry Potter and the Prisoner of Azkaban defined the series in a way no one expected.

Here’s how the Harry Potter series got legit.

The Successor

[Credit: Warner Bros.]

Turning J.K. Rowling’s first novels into movies was an arduous process. Even before he was done with the second movie, an exhausted Chris Columbus decided to move on rather than dive back into directing the third film.

Several names were considered to succeed him – and several directors considered whether or not to commit to the project. Guillermo del Toro turned down the opportunity, as did M. Night Shyamalan. Divine Secrets of the Ya-Ya Sisterhood filmmaker Callie Khouri was a contender, as was Kenneth Branagh, who at the time was playing a role in Chamber of Secrets.

In July of 2002, while Columbus was still shooting the second movie, Warner Bros. announced Alfonso Cuarón as the director of Harry Potter and the Prisoner of Azkaban. This was not an obvious choice. Cuarón’s most recent film was Y Tu Mama Tambien, an explicitly sexual art film. Fortunately, J.K. Rowling was a fan. He had also directed A Little Princess for Warner Bros., which is a great movie too often overlooked.

And while the first two films were released only a year apart, in 2001 and 2002, Warner Bros. decided to push the release of Azkaban from 2003 to the following May. That gave Cuarón and the team more time, and also meant the actors would be just that bit older in the movie, which turned out to be a smart choice.

Change Is Good

[Credit: Warner Bros.]

Cuarón used the extra production time to create the most distinctive Harry Potter film yet. Step one was choosing cinematographer Michael Seresin to help bring the new stage of Harry’s world to life. Seresin was best known at that point for his extensive collaboration with director Alan Parker. Their films, including Midnight Express, Angel Heart, and Fame, felt very real even when the events edged into the fantastic.

The two chose to shoot a great deal of Prisoner of Azkaban on actual locations rather than stages. That gave the movie an immediately recognizable air of reality – and also a slightly strange atmosphere, as Harry’s surroundings were never quite “normal.”

Just look at one of the first big setpieces in Prisoner of Azkaban. Harry flees the Dursley home and ends up forlorn in a park, only to be “rescued” by the Knight Bus. The scene is melancholy and a bit unpredictable, but then the bus speeds in, bringing a bizarre comic element. The look and feel give the whole sequence more impact. The same goes for scenes with Hagrid shot in real forests and many other scenes where effects and recognizable locations combine to create Harry’s unique reality.

Darkness Comes Alive

[Credit: Warner Bros.]

Cuarón and Seresin’s use of real locations helps emphasize the darkness that encroaches upon Harry and friends. Threats get more real in this movie; there’s a new sense of unease. As Harry learns more about his parents, and more about the Wizarding World, the stakes start to become truly overwhelming.

And Voldemort gets a proper introduction in Harry Potter and the Goblet of Fire. Sure, we saw Voldemort’s very CG face in the first film, and Tom Riddle shows up in Chamber of Secrets. But Mike Newell got to cast Ralph Fiennes in the role for Goblet of Fire. Fiennes brings a slightly operatic but very believable air to the dark lord when his Earthly form is reconstituted. His introduction – along with the appearance of Voldemort’s enforcers, the Death Eaters – makes Goblet of Fire just as important to the establishment of the story’s sense of danger as Prisoner of Azkaban.

So Much Material

[Credit: Warner Bros.]

While Prisoner of Azkaban had to deal with a directorial change, that wasn’t the only challenge for the fourth movie. Cuarón was only willing to direct one film. Just as his movie was about to open, Mike Newell got to work on the fourth movie.

The fourth book is almost twice as long as the third, which meant that characters and scenes had to be greatly condensed. At the time, Goblet of Fire had the most noticeable changes from book to film. That set a pattern for the films that followed, as Rowling’s books got longer, and more dense. Even with the changes and abbreviations, Cuarón and Newell had three great talents to rely on: Daniel Radcliffe, Emma Watson, and Rupert Grint.

Hogwarts Champions

[Credit: Warner Bros.]

This lead trio truly carries both films, as the rapidly-maturing actors demonstrated more confidence in their roles. There’s a pronounced jump forward from the characters as they’re seen in Chamber of Secrets to the people we see in Prisoner of Azkaban.

By the end of Goblet of Fire, the actors have made the roles utterly their own. While the characters begin to diverge slightly from the books, they still feel like Rowling’s creations. This is evident in Goblet of Fire, in which the actors increasing abilities overcomes any problems that might have arisen thanks to the condensed material.

The producers and Alfonso Cuarón also had to accept the death of Richard Harris, who played Dumbledore in the first two films. They chose Michael Gambon to play the role going forward; he would prove to be yet another brilliant choice. Gambon’s Dumbledore is significantly different from Harris’s, but he’s appropriate to the darker, more worldly tone of these movies.

Harry Potter and the Sorcerer’s Stone and Harry Potter and the Chamber of Secrets turned readers into movie fans. They proved beyond any doubt that Rowling’s books could work on the big screen. Alfonso Cuarón and Mike Newell did something even more impressive. They proved the material was as legitimate as any other story, and that the Harry Potter films should be taken seriously.