The 237,000-square-foot building, known as Le Freeport, opened its 10-ton doors, joining a global network of impenetrable warehouses where taxes are suspended and art can be stored and sold away from prying eyes.

As the international art market has boomed, so has demand for such discreet storage, particularly for costly art picked up as an investment and never intended for a living room wall. But a year after Luxembourg’s Le Freeport opened at a cost of nearly $60 million, the Monets and Picassos have been slow to arrive.

The reason: the protracted public-relations disaster of Le Freeport’s biggest investor, Swiss entrepreneur Yves Bouvier. Mr. Bouvier was briefly jailed in Monaco earlier this year, after he was accused of selling paintings to Russian billionaire Dmitry Rybolovlev at inflated prices.

On Sept. 14, Mr. Bouvier faced new charges. He was questioned in France by authorities investigating allegations by Catherine Hutin-Blay, the stepdaughter of Pablo Picasso, that works in her collection were pilfered and sold to Mr. Rybolovlev. Mr. Bouvier was freed after posting a €27 million ($30.5 million) bond.

In a statement, Mr. Bouvier denied the allegations in France and said the pieces in question were properly purchased on behalf of his Russian client. Some were meant for the walls of a chalet owned by Mr. Rybolovlev, while others, including the Picasso painting “Espagnole à l’éventail,” were deposited in a freeport in Geneva, he said. In an interview with The Wall Street Journal in Geneva earlier this year, Mr. Bouvier also denied the allegations of defrauding Mr. Rybolovlev.

Mr. Rybolovlev’s attorney said authorities were free to examine works in his collection believed to have been stolen. Mr. Rybolovlev hasn’t been accused of wrongdoing.

The legal issues have amplified the challenges facing Luxembourg’s Le Freeport, which was already contending with a chorus of critics who say the opaque nature of freeports makes them susceptible to illicit activity such as money laundering or harboring stolen goods. Earlier this year, lawmakers in Luxembourg responded to those concerns by bolstering the country’s anti-money-laundering laws to specifically apply to firms operating at Le Freeport—which now have a heightened legal responsibility to know what they are putting into storage and for whom.

Mr. Arendt, Le Freeport’s managing director, said the facility has safeguards to prevent illegal behavior and that he is confident Le Freeport isn’t used for money laundering. He also said the facility can adapt to the legal change—and perhaps even use it as a selling point for “honest art collectors.”

But Mr. Bouvier’s travails have hurt business. “We’re behind on our objectives, as a result of all of this mess,” Mr. Arendt said in an interview in May, as he sat in an empty showroom at Le Freeport. On Wednesday, Mr. Arendt sounded hopeful that the bad publicity had faded. “We’re more or less on track,” he said, though “we’ve had a lot of explaining to do.” Mr. Arendt said Mr. Bouvier has no role in running the freeport.

In the art market, “reputation is pretty much all you have,” said Harco van den Oever, who runs London-based Overstone Art Services.

About 60% of Le Freeport’s storage space has been rented to the firms that deal directly with collectors and store their art at the building, according to Mr. Arendt. The biggest of those firms, FineArt Logistics Natural Le Coultre SA, which accounts for roughly half of that rented space, is Mr. Bouvier’s company, and has so far only filled about 15% of its available square footage.

Freeports are an outgrowth of bonded warehouses, where commodities like grain in transit across borders could be stored, without incurring taxes. There are dozens of such warehouses around the world.

A few, like Le Freeport, are tailored to the needs of high-end art collectors. At Le Freeport in Luxembourg, artwork that remains inside its walls is exempt from value-added tax, which can be as high as 27% in Europe, and from customs duty. Art sales within the building are also tax-free. Art can remain in a freeport for years, through multiple transfers of ownership. Firms offering art-related services like restoration and financing set up shop on the grounds.

The value of the global art market reached a record €51 billion last year, according to a recent report published by the European Fine Art Foundation, the organizer of a major annual European art fair.

Nearly a third of art collectors and professionals surveyed last year by Deloitte Touche Tohmatsu Ltd. said they had used a freeport. Experts speculate that the freeport in Geneva, which traces its roots back to 1854 and is controlled by local authorities, may house the most valuable art collection in the world—although its holdings are generally secret.

Proponents say freeports allow art to be moved around the world without incurring punitive taxes. Critics say the warehouses can invite fraud. “I can’t see any better way for people to launder money than to go through a freeport,” says James Palmer, founder of Mondex Corp., which specializes in recovering art looted from its rightful owners.

Someone interested in laundering money could arrange to use ill-gotten funds to purchase a painting inside of a freeport, Mr. Palmer said, and then resell the painting inside of the facility a few months later. Experts say it can be difficult for a customs official relying on a printed summary of a sale conducted inside a freeport to know for sure if the amount exchanged was appropriate, or if a painting that was purportedly sold was real.

U.S. authorities are also concerned. Authorities “really don’t know what’s being stored there and who really owns things,” said Daniel Brazier, a special agent with Homeland Security Investigations, an arm of the U.S. Department of Homeland Security, at a panel discussion on freeports in New York in June.

Customs officers on site at Le Freeport open and inspect “almost every box” that arrives, Mr. Arendt says. Sylvie Atten-Liber, deputy director at the Luxembourg Customs Office, said officers at the facility check arriving artworks against databases of stolen pieces and have found no stolen items.

In Switzerland, the government has been urged by the Federal Audit Office to bolster the presence of customs officials at all of the country’s freeports and is expected to disclose plans by the end of this year. The new chairman at the Geneva freeport has a mandate for a “cultural revolution” involving increased transparency, said Patrick Baud-Lavigne, an official with the canton of Geneva’s department of security and economic affairs.

In 1995, Swiss authorities discovered a trove of smuggled artifacts, including Etruscan bronzes, in the Geneva freeport. In 2010, a Roman sarcophagus suspected of having been looted was discovered at the Geneva facility. A spokeswoman at the Geneva freeport didn’t respond to a request for comment.

Luxembourg, a tiny grand duchy surrounded by Germany, France and Belgium, has constructed an entire industry around privacy and low taxes. In 2008, consultants at Deloitte devised a plan to make Luxembourg “an art and finance cluster” built around a freeport, said Deloitte Director Adriano Picinati di Torcello. A Luxembourg government spokesman said the facility was intended to contribute to the “diversification of the economy.”

Luxembourg turned to Mr. Bouvier. The 52-year-old had transformed his family’s sleepy, Swiss logistics firm into a go-to outfit for shipping valuable art around the world. He had invested in Geneva’s freeport, and opened a Le Freeport in Singapore, in 2010, helping to earn him the moniker “the Freeport king.” That year, he got involved in the Luxembourg project.

In September 2014, Luxembourg’s Grand Duke and several local government ministers were among the guests mingling at Le Freeport’s opening ceremony. An orchestra performed a piece of music composed specially for the event.

But the mood soon turned flat. In February, Mr. Bouvier was arrested in Monaco. The previous month, a family trust tied to Mr. Rybolovlev, the Russian client, had filed a complaint alleging that Mr. Bouvier overcharged him for paintings, while acting as an art-dealing middleman.

Mr. Rybolovlev made his fortune by building Uralkali, a giant producer of potash, a key fertilizer ingredient. Mr. Bouvier said he sold Mr. Rybolovlev roughly $2 billion worth of art over the years, including Picassos and Rothkos.

Mr. Bouvier dismissed the allegations. “It’s the market that makes the price,” he said. His attorney, David Bitton, said his first court date in Monaco is this week..

A spokesman for Mr. Rybolovlev declined to make him available to comment.

Mr. Bouvier stepped back from his regular duties as a Le Freeport board member in April. He said that banks and investors are now hesitant to lend to him, and he said he has halted several freeport projects in locations including Dubai and South Korea.

Mr. Bitton said his client’s Le Freeport facility in Singapore, which opened in 2010, has also seen a “negative business impact” as a result of adverse publicity, though he declined to elaborate. Tony Reynard, Le Freeport Singapore’s chairman, didn’t respond to requests for comment.

Jennifer Smith contributed to this article.