Describing the talks as “serious, straightforward, businesslike, candid discussions,” Ms. Sherman said that it remained unclear if Russia was prepared to reduce its military buildup near Ukraine following the talks—or even whether Moscow was committed to stay at the negotiating table.

“We have a long way to go,” she said.

Deputy Russian Foreign Minister

Sergei Ryabkov

described the talks as “difficult, long, very professional, specific, without any attempts to embellish anything, to get around any sharp corners.”

“A businesslike, professional conversation in itself sets you up for an optimistic mood,” Mr. Ryabkov added. “But the main questions are on hold and we do not see the American side’s readiness to resolve them in a way that suits us.”

Monday’s talks, and two more rounds later in the week with a broader set of European countries, take place in the backdrop of a massive buildup of Russian troops along Russia’s border with Ukraine. Moscow says the troop movements are a response to a threat to its own security from the West. The Kremlin has accused the North Atlantic Treaty Organization of trying to encroach into Russia’s backyard with military ties to Ukraine. It has demanded that NATO halt its outreach to eastern countries including Ukraine.

“It is in this area that we have the greatest friction with the United States,” Mr. Ryabkov said. “It is very important that Ukraine can never join NATO in the future. Neither Ukraine nor Georgia should become members of the North Atlantic Alliance. We need iron, legal obligations, not promises, but guarantees.”

Mr. Ryabkov said his side told the U.S. diplomats, “We have no intention to attack Ukraine. There is no reason to fear any kind of escalation scenario.”

The Biden administration is trying to defuse tensions by offering Russia negotiations on intermediate-range missiles in Europe and proposals to scale back military exercises on a reciprocal basis.

Ms. Sherman said that the U.S. had offered to meet again with Russian officials on security issues but didn’t say if Moscow had agreed to continue discussions.

“We will not allow anyone to slam closed NATO’s open-door policy,” Ms. Sherman said, adding that the two sides had a “frank and forthright discussion” about issues including military exercises and missile placement in Europe.

The talks began Sunday night with a working dinner between Ms. Sherman and Mr. Ryabkov. They were accompanied by senior military officials: Lt. Gen.

James Mingus

for the U.S. and Russia’s deputy minister of defense, Col. Gen.

Aleksandr Fomin.

Russian officials and representatives from NATO nations are scheduled to meet in Brussels on Wednesday. On Thursday, discussions will be held in Vienna at the Organization for Security and Cooperation in Europe.

President

Biden

has ruled out military intervention on behalf of Ukraine. But administration officials say they are ready to impose major economic costs on Russia and step up military assistance to Ukraine if Russia further violates the country’s sovereignty or territorial integrity and. Moscow annexed Crimea from Ukraine in 2014 and has backed separatists in the country’s east.

Washington is unlikely to impose the biggest energy or financial sanctions that would cut Russia—a top energy supplier for Europe—off from its customers or seal off the country’s financial system, according to people familiar with the matter.

Instead, officials are looking at more targeted measures, including erecting export barriers to block international sales to Russia of products with a certain percentage of American content, as well as preventing Moscow from getting access to cutting-edge microchips used in everything from aircraft to consumer electronics, according to people familiar with the matter.

Asked about whether U.S. allies would support imposing export controls on goods to Russia, Ms. Sherman said, “We are finding a lot of understanding, agreement and interest in pursuing them.”

Mr. Ryabkov said the West had only “sanctions and blackmail” left in its foreign policy tool kit, but said he didn’t see “the situation as hopeless.”

“For the first time we were able to talk about issues that were previously invisible and existed, but as if behind the scenes,” he said. “And now things are called by their proper names. And this has a healing effect on our relations with the West.”

Write to Michael R. Gordon at michael.gordon@wsj.com, William Mauldin at william.mauldin@wsj.com and Ann M. Simmons at ann.simmons@wsj.com