All writing

Simon James and James Simon, for web developing

So, during my early years in cybersecurity, there was a tool named HTTrack. This tool simply copied websites. Because it wasn't so popular in the circles I knew, and had one of the most complicated interfaces a tool could have, I, as a rookie, saw it as magic, and the people who could actually use it as superhumans!

I was going to be very disappointed when I used it for the first time.

Reasons? Extremely complicated for no reason, with barely any functionality that satisfied my needs! You heard it right. Because this tool was very old, it struggled to copy the modern, dynamic interfaces of websites. And when I wanted to try it on a static website, IT COULDN'T EVEN COPY IT EXACTLY AS IT WAS!

It was like a spear: useful against mammoths, BUT NOT TANKS!

Knowing that I needed to both crawl a website and copy each page, then put everything into a folder properly to reconstruct it, I wanted a good crawler first.

Guess what? I ALREADY HAVE ONE!

Integrating WebCrawler into this workflow was one of the most satisfying things I've done in engineering. I already had a tool capable of discovering pages, following links, and organizing the results. Instead of writing another crawler from scratch, I adapted it into the foundation of DoppelGNG.me, letting it handle website discovery while the new system took care of capturing, reconstructing, and validating the actual contents.

I also used headless Waterfox to interact with the websites as a browser would. This allowed me to work not only with the original HTML responses but also with rendered pages and the network activity generated while loading them.

The workflow had to be simple, right? Copy each piece of content, make a file, and repeat until you reach the end.

Not for a guy who was disappointed with HTTrack.

I established a bidirectional communication channel between the browser and DoppelGNG.me using WebDriver BiDi. Instead of relying solely on the contents retrieved through ordinary HTTP requests, the system could observe browser network events and collect the responses associated with them.

This is much like listening to the pulse of a creature. You aren't just looking at what it looks like; you're also observing what happens inside while it is functioning.

By doing so, it became easier to capture not only the appearance of a website but also the resources and browser-visible behavior that made it work.

Of course, collecting everything was only half the problem. Modern websites aren't simply collections of independent HTML files. They depend on JavaScript, CSS, fonts, images, dynamically requested resources, and references connecting all of them. Copying those files without preserving their relationships could leave you with a folder full of resources and a website that still wouldn't work.

So, I added dependency discovery and resource rewriting. DoppelGNG.me could identify the resources a page depended on, preserve the original responses separately, and rewrite the necessary references to point toward their reconstructed local counterparts. Resources from external origins could also receive local aliases, allowing captured assets to be served without relying on their original locations.

The result wasn't merely a folder of downloaded files. Each completed capture became a standalone project with its own local replay server, capable of serving the archived frontend.

But this is not the end.

Remember the copying issue with static websites? I also addressed it for dynamic ones, checking and comparing the reconstructed pages against the original website rather than assuming that successfully downloading the files meant the job was done.

I added automated validation for both desktop and mobile viewports, comparing screenshots taken from the top, middle, and bottom of each page. But screenshots alone weren't enough. The system also checked text, headings, fonts, page dimensions, images, links, browser errors, and navigation between internal pages.

I even included checks for supported interactive elements, so that a button or disclosure control couldn't simply look correct while doing absolutely nothing!

And, because I didn't want one failed operation to force the entire process to restart, I added checkpoints, saved manifests, and resumable captures. Previously verified resources could be reused, while incomplete or failed work could be attempted again.

The system wouldn't declare a capture complete until the required reconstruction and validation checks passed. If something failed, it had to remain incomplete rather than pretending that everything was fine.

Sounds very cool.

But there was still one thing I wanted to verify: could DoppelGNG.me actually reconstruct a real website and run its captured frontend independently of the original?

So, on October 17th, I tested it against my own portfolio, emilvaliyev.com.

The recorded test captured 19 discovered routes and 110 resource records. It completed 72 page-rendering observations across desktop and mobile viewports, alongside 80 internal navigation checks. I also tested the exported copy in a fresh browser with access to the original website blocked, and the recorded checks passed.

That was exactly the kind of result I had wanted when I first encountered HTTrack.

Of course, DoppelGNG.me doesn't magically recover a website's backend source code, database, or every possible behavior. It reconstructs the frontend resources and states it can observe and verify. Features relying on unavailable backend services may still require additional work.

But that distinction matters. I didn't want a tool that merely claimed to copy websites. I wanted one that could show what it had captured, what it had reconstructed, what it had verified, and what remained incomplete.

And I wanted the result to actually work.

What once looked like magic to me as a rookie had become an engineering problem I could finally approach myself.

Apparently, the people using HTTrack weren't superhumans after all!

All writing