The enforcement of vices on the Internet
Drugs, sex, and economics: the enforcement of vices on the Internet is often a tale of futility. But surely something is better than nothing?
Source: https://www.nytimes.com/2012/03/18/nyregion/from-roosevelts-new-york-lessons-in-the-futility-of-policing-vice-in-the-city.html
I’ve been looking at how people do bad things on the internet for a while. This has taken me to various communities, forums, server, chats, and other digital hangout spots where I can observe and read peoples’ behavior. This boots-on-the-ground approach helps me get a pulse of why people do what they do, how, and get a sense of what comes next. These are all useful things for policies or so called “scientific interventions.” Essentially, we want to figure out how to stop people from doing bad things, like generating and sharing synthetic nonconsensual images (”deepfake pornography” for the masses). The goal of this blog post is to try to understand our Fiverr findings by thinking about them in the same terms as other vices on the Internet.
We recently published a paper that showed how there are listings on Fiverr (paper; blog post), which is a popular gig work platform, that allows people to hire contractors to create pornographic deepfakes. That is, you can hire a freelancer to help you make AI-generated nudes of someone else on Fiverr, same as you would hire someone to help you make a logo or run online ads. When Mohamed (PhD student who led the Fiverr work) approached me and told me about these services, it felt important to document it, not just because it’s a thing that is happening, but because it tells a broader story of existing enforcement efforts.
Source: Fiverr paper linked above.
The story of drugs online provides a useful way of thinking about enforcement. During my PhD I looked at how and where people sold illicit substances online. “Darkweb marketplaces” (i.e., eBay-like websites that you access through Tor) were already mature enterprises in the 2020s, accounting for billions of dollars in revenue. They had come a long way from the moment that someone had the straightforward idea of selling drugs over this then-new thing called “the Internet.” If we trace the evolution of drug sales online, we can probably get a blueprint of what will play out with deepfake pornography.
First came the people who advertised directly on websites: “canna seeds for sale, we take western union.” Then, came the forums (e.g., “The Hive”). Afterwards, we got the first semblance of markets (e.g., “Adamflowers” then “The Farmer’s Market”). As you may imagine, the DEA, FBI, and adjacent acronyms cracked down on these sites aggressively. Eventually, came the “Silk Road” and the darkweb era facilitated by the emergence of cryptocurrencies. The way in which technological developments (like Tor and cryptocurrencies) ushered new opportunities to sell narcotics and other illegal goods is fascinating. We will tell, however, that story another day. Today, we will talk about the market pressures that pushed these websites to evolve.
In his thesis, “Outsourcing Cybercrime”, Rolf van Wegberg [1] draws from transaction cost economics to describe how cybercriminal markets increasingly become more sophisticated. One way they do so is by becoming more structured. This is most clear when you look at forums that then become marketplaces. For example, before Facebook Marketplace was a thing people used Facebook Groups like “Buy/Sell/Trade New York” to sell things, and before that, people directly posted on their profile that they were selling something. The transition from sales posts in personal profiles, to groups dedicated to buying and selling, and then a dedicated platform of sales illustrates how markets seek structure and sophistication. The same pattern plays out for drug sales: standalone websites advertising drugs, then forums, then marketplaces.
Source: https://www.aljazeera.com/news/2013/11/6/silk-road-illicit-online-market-reopens
Markets are good for vendors and buyers. A market-type-of-website makes it easier to search for products, it introduces reputation systems (e.g., reviews and ratings), standardizes payments, and introduces a mediator who can resolve disputes when things go wrong. There is also an incentive for people to create the market itself. If you are the owner of the market, you take a cut of all the deals. In the Silk Road trial, the FBI said it had seized 144,000 BTC (yes, a hundred and forty four thousand Bitcoins) from Ross Ulbricht. In 2022, the Hydra market was allegedly pulling 1 billion sovereign American dollars. Owning the platform is definitely where the money is at, so when a market disappears, another quickly steps in. The demand for drugs definitely does not go away. And you, as the market owner: all you need to do is avoid the cops and the suits [2].
Source: https://www.nbcnews.com/id/wbna6809149
But before starting a market there are alternatives to explore. Starting a market is tedious. Before LLMs, you needed at least some semblance of coding experience to get it up and running. You then needed to overcome a cold start problem and attract vendors and buyers. So instead, people tested existing markets like eBay. Vendors would advertise the sale of drug “catalogs” or “magazines” while making air quotes––”it would be a shame if someone happened to accidentally drop a vial with four carbon rings, some oxygens, and a few hydrogens originally meant for veterinary use but which gives you superior athletic performance, right?” With less frills, people also just directly sold cannabis on Facebook Marketplace.
If this post is clear, by now you should see the parallels with Fiverr.
As of last year, there were three main places where you could go to get your fix of synthetic nudity. Dedicated websites where you could upload images and perversely manipulate them yourself (e.g., nudifiers). Forums where people share their creations, commission new fakes, and share tips and tutorials (e.g. MrDeepfakes). And these part-forum, part-cloud-infrastructure, part-market platforms where you could train and share AI models, share your images/videos, and earn platform currency that enables you to generate more images/videos (e.g., CivitAI). This last one is not yet a market like eBay but has a lot of market-like characteristics. For example, you earn platform currency based on how much people interact with your model, which in turn makes model providers (think of them as vendors) to try to put out better models (think products) to attract more users (customers), like a market [3]. Given all of this, it is not surprising that people are testing the waters in established markets, like Fiverr. Economics and market dynamics demand more sophistication and structure.
The market for deepfake pornography is still finding its footing. Many peddlers of artificially generated genitalia rely on ad-hoc mechanisms to monetize their creations: they have tip jars on their profile, they create private communities and charge for access, or they rely on affiliate programs to nudification websites and such. Increasingly, though, people are trying their luck in established markets. Fiverr, OnlyFans, Fanvue, and others, have structure and sophistication. They make payments easy. They have brand reputation. They have active user bases. And they are not turning these vendors away. How long do you think anabolic steroid vendors stayed active on eBay before they were shut down?
There are many similarities in the trajectories of online drug sales and synthetically generated body transgressions. The websites, the forums, the economic pressures. These factors help us understand why these services are popping up on gig platforms and mainstream markets. But ultimately there is one crucial difference: making and commissioning deepfakes is pretty cheap. Websites with very powerful models give daily free credits for generation. For users with GPUs, they can generate so much of this stuff and they take commissions for free and with gusto. There is no incentive yet to create a marketplace for this stuff since so much of the volume is gratis!
Source: <redacted>
The low cost of production also means enforcement faces a different challenge here than it did with drugs. You cannot cut off the money if there is no money to cut off. Procuring sexually explicit videos of your colleagues, classmates, or favorite celebrity is cheap, easy to make, and easier to find. In fact, the listings we identified (and which we reported) in our Fiverr study still seem to be there. Chasing down deepfake pornographers seems like hopeless work––for every head you cut off, a new one sprouts. And then, even if you get rid of these websites on the clear web, they can migrate to encrypted chat groups or the ~dark~ web. But hear me out. One, we are a long way from that, and if these services migrate to shadier corners we will at least have a sense that enforcement is actually doing something. Two, we pursue CSAM distributors relentlessly, even despite all the constraints above. And three, come on, detecting these listings on a website like Fiverr is trivial in 2026.
If we ever see these services migrate to the shadier parts of the Internet, and if a market ever pops up exclusively dedicated to the commercialization of deepfake pornography, as bleak as it sounds, we will know we are on the right track.
Related Reading
Plug and Prey? Measuring the Commoditization of Cybercrime via Online Anonymous Markets
Deepfakes on Demand: The rise of accessible non-consensual deepfake image generators
Exploring the Use of Abusive Generative AI Models on Civitai
Footnotes
[1] Rolf was on my thesis committee and has been a phenomenal mentor. Thanks Rolf!
[2] Fun fact, the IRS is even more interested than the FBI in whether you own the marke: https://home.treasury.gov/news/press-releases/jy0701
[3] It is important to note that CivitAI cleaned house after substantial pressure from payment processors; they introduced cryptocurrencies and a gift-card-type of payment. There was a genuine reluctance to see Visa and Mastercard go away which means that the alternatives are just not as good.