GOP Rep. Doug LaMalfa of California, a longtime McCarthy ally who voted for McCarthy on the first ballot, told reporters he will vote for Jordan on the second ballot.

“I’m not against him,” LaMalfa said of Jordan. “I was for Kevin McCarthy the whole time and I thought the process has been terrible, what has happened to him and the things that he’s been held up for. So I’m voting for Jim Jordan because he’s a good guy. He’s done good work on committee. And we need to move forward with this place today, and get our work done.”

Others said they cannot be moved. GOP Rep. Carlos Gimenez of Florida, who voted for McCarthy, said he will never vote for Jordan but would consider voting for other candidates.

“I think we need to move forward as a conference and find somebody that we can all coalesce around,” Gimenez said.

A group of centrist New York Republicans – a key bloc of lawmakers in the House, who helped hand the GOP the majority – privately discussed their vote for speaker for weeks, conferring with one another and weighing the idea of voting as a bloc, sources tell CNN.

As they wrestled with whether to back Jordan, one thing that came up repeatedly and gave them pause: Jordan’s votes against key priorities in New York, including voting against aid for superstorm Sandy and against 9/11 health care funding for first responders.

But they didn’t make a final decision on how they would vote until Tuesday morning, sources said. Four voted against Jordan.

Jordan’s opponents angry with House process

On Monday, several key holdouts said they would support Jordan, including Rep. Ann Wagner of Missouri, who had previously called Jordan a “nonstarter.”

“I feel like he can bring together everybody, from the moderates to the ultra conservatives, and Republicans across the spectrum,” said Rep. Nicole Malliotakis of New York. “The bottom line is we’ve got to get back to work. We don’t have any time here to waste.”

But there remains a group of lawmakers publicly opposing Jordan, including GOP lawmakers still angry that a small group of Republicans forced out McCarthy and then opposed the speaker nomination of Scalise, who initially defeated Jordan inside the GOP conference, 113 to 99.

“I can’t get past the fact that a small group in our conference violated the rules to get rid of Kevin, and then blocked Steve,” said Rep. Don Bacon of Nebraska, who voted for McCarthy on the first ballot. “You don’t have a process where I play by the rules and these other people can’t and then they get what they want. That’s not American. Americans want fair play and rule of law.”

House Democrats have blasted Republicans for putting forward Jordan as the next potential speaker. “I was the last person on the floor January 6, and the idea that this guy is the Republican nominee to be speaker, a guy who aggressively agitated the activities that happened on January 6, I think is disgusting,” said Rep. Jim McGovern, a Massachusetts Democrat.

Jordan’s backers have urged the conference to unify around him – even those who went after McCarthy and opposed Scalise.

Rep. Scott Perry of Pennsylvania argued that those angry shouldn’t take it out on Jordan because he supported both McCarthy and Scalise.

“Feelings are hurt,” Perry said. “But Jim didn’t have anything to do with that. So they need to assign their ire, if you will, to those who they think deserve it – but certainly not Jim Jordan.”

This story and headline have been updated with additional developments.

CNN’s Lauren Fox, Kristin Wilson, Morgan Rimmer, Clare Foran and Sam Fossum contributed to this report.