Was Google Earth Stolen? (no)
[October, 2021]
I recently watched “The Billion Dollar Code” limited-series on Netflix, which claims that Google Earth owes its existence to a German project called TerraVision, created by the art/tech collective ART+COM. The show chronicles their interactions and the ultimate lawsuit against Google for a billion dollars, which — spoiler alert — failed.
I am personally drawn to any stories of inventors having their work stolen by greedy corporate hustlers. I can relate to the inventors in this case and their small company struggles. I truly respect the early and innovative work done by ART+COM. If anyone is owed credit or money for their invention, I certainly want them get it.
However, as a co-founder of the startup that built the original EarthViewer PC app in 1999, later sold to Google and rebranded “Google Earth” — and as one of the people who wrote many of the bits they alleged were stolen, I am in a pretty good position to call bullshit on this.
I wrote a highly relevant blog post in 2007 called “How Google Earth [Really] Works,” which explains in simple terms what’s unique about Google Earth and its key patents. It’s a bit more technical than this article, but anyone trying to deeply understand “how does Google Earth really work?” (or where did it come?) from should ideally read that first.
Now, I’d never even heard of ART+COM or TerraVision until about a month ago (I was notified just before the Netflix show came out). I wasn’t contacted regarding the series, the lawsuit, or the “making of” episode featuring real participants in this story. I was mildly aware of an earlier IP infringement suit by a different company. FWIW, that lawsuit also failed.
At Keyhole, I’d worked with the two brilliant real people who were named in the actual ART+COM lawsuit. They were merged into a single fictional “Brian Andersson” character in the TV drama. I imagine this was to minimize charges of defamation. However, one of them is actually named Brian and the other sadly passed away this year. For the ART+COM side of the story, the series mashed up two more fictional characters: “Juri Müller” and “Carsten Schlüter” as the Germans involved. The “Making Of” special episode highlights four or five real people with interviews. But it remarkably only covers one point of view.
Let’s talk indisputable facts for a moment. ART+COM filed its (now-invalidated) patent in December 1995. That patent was the basis of the lawsuit against Google almost two decades later. However, one month before that filing, Silicon Graphics (SGI) filed another patent it called “Clip Mapping,” which supports high quality whole-earth 3D rendering (see the blog post above). Google Earth uses an even more advanced form of “clip mapping” that goes beyond what SGI had invented. That better algorithm ran on ordinary PCs in 1999 and today runs in your browser.
The timing of these two patents suggests one of two things is true:
- ART+COM met with SGI and inspired a devious SGI executive (or two) to schmooze them until they revealed their secret algorithm. They then flew back to Germany, while certain SGI employees flew back in time to begin work on a superior technique that was tied to and sold exclusively for their very expensive 3D graphics hardware. The time travelers cleverly filed their patent first, at least chronologically (wink).
- Or, perhaps SGI was already rendering high-quality 3D earths before they ever met with ART+COM. This would have been a natural result of decades of work on flight simulators. Let’s also assume that SGI quietly provided hardware to top-secret NSA and NIMA US security analysts. A senior manager (such as “Brian”) might know about such deals, but couldn’t disclose it to ART+COM. So when “Brian” saw ART-COM’s cool public-facing 3D planet demo, he/they invited the Germans to party — why? To get them drunk and steal their secrets? No. To recruit them to join their team at SGI. That explains “Brian’s” effusive praise and why the German duo argued over moving to California, as depicted in the series.
If the answer is not obvious enough yet, SGI built and showed their own public-facing demo called “From Space to [Your] Face” that smoothly zoomed in from seeing the entire planet, down to a spot in the Alps. They did this to show off their new InfiniteReality hardware with their custom hardware clip-mapping feature. The same demo later zoomed all the way from space down to a scale model of a Nintendo-64 console, which SGI had quietly helped create as another secret side project starting around 1993.
AFAIK, SGI did this all without any ideas or code from ART+COM. If SGI didn’t infringe on ART+COM’s patent back in 1996, then no one did. FYI, Microsoft later bought most of SGI’s intellectual property when it went out of business later, including this famous clip-map patent.
From SGI to Intrinsic Graphics
Chris Tanner, along with one of the two real “Brians,” had originally invented the “Clip Mapping” algorithm at SGI in the 90s. That first solution used this special SGI hardware to solve one hard problem, limiting it to SGI supercomputers only. However, after leaving SGI in 1997/98, Chris came up with a novel way to solve this that could work on ordinary cheap consumer PC 3D graphics cards. This new solution earned its own patent, now branded as “Universal Texture.” (also explained in my blog post above)
That, plus new code that I and others added, is what enabled Keyhole to so quickly launch an amazing PC-based EarthViewer app in 2001. PC-based deployment is a hard problem that ART+COM apparently never solved.
I joined Chris in 1998 as employee #2 of Intrinsic Graphics, where he, Remi Arnaud, Brian McClendon and Michael Jones were the four co-founders. I didn’t have to work in a garage at that point. We’d rented a converted post-office in downtown Mountain View. Then in 1999 a few of us co-founded Keyhole as an internal spin-off with John Hanke joining us as CEO.
Let’s be totally clear. I wrote my part of the codebase from scratch in that office. The only thing we retained from earlier prototypes or demos was Chris’s now-patented Universal Texture code, which became part of our 3D game engine, written entirely in C++. This constituted the first version of what later became “Google Earth,” called “EarthViewer.”
The Netflix series depicted our actual app and UI design from the original v1 EarthViewer product, but presented it as if it came out much later from Google. That’s a bizarre thing to get wrong. Earthviewer was available for free from 2001/2002 on. The company was acquired by Google in 2004. FYI, I actually left Keyhole in 2001 and never joined Google.
By the 2010s, the source code for Google Earth was evidently rewritten entirely in Javascript to use WebGL and run in ordinary web browsers. This was a long-time goal of ours to lower installation “friction” and make the app more universal. The lawsuit apparently refers to the javascript code, which anyone could have analyzed with a web browser.
The problems ART+COM did solve are worthy of praise, but are too obvious to patent or protect IMO, even in the 90s. They are what any good engineer would likely pick from available software tools.
For example, quad-trees were invented in the 1970s (1966 if you consider morton-codes). The key idea is to subdivide a large map or image into 4 equal tiles, recursively. Each tile or “node” also gets subdivided into 4, repeating as needed. As you zoom into the planet, you begin with the “root node” or main tile, and then the algorithm finds smaller tiles with effectively higher and higher resolution as needed.
Clip-mapping and Universal Texture both assume quad-tree subdivision of the source data, as do many other common uses, like storing roads and labels. I came up with a few helpful optimizations that improved efficiency here over the basic approach.
The “addressing scheme” is also something ART+COM lawyers claimed was obviously copied by Google. In reality, my colleagues and I agreed on the simplest approach. Each node in a quadtree has four child nodes that could be numbered #0, #1, #2, #3 (00, 01, 10, 11 in binary). So the “address” of any given node is just the string of these binary codes from the root of the tree and out to the smallest child node. It’s basically morton codes again.
To be clear, Chris Tanner’s “Universal Texture” is what managed the earth texture efficiently on a PC, and that was indeed novel. John Carmack later announced “Mega Texture” that worked along very similar lines. No one claimed Carmack stole our ideas, though he did visit Keyhole and got early demos. Good engineers write their own code.
The idea that “Brian,” while being a director of hundreds of people on the Google Geo team, took this working, novel, and patented Google Earth code that Google had already bought for millions of dollars and re-wrote it to match what “he” may have vaguely remembered seeing in ART-COM’s 1995 demo — is so totally absurd as to question the basic causality of the universe.
At early Keyhole, I improved on Chris’s work by adding image compression and streaming. Chikai Ohazama and Phil Keslin built the full network stack and made many more significant optimizations on the server. Mark Aubin, another Keyhole co-founder, attacked the problem of efficiently assembling many terabytes of source maps into one big map (see ortho-rectification, clip-gen) before we chopped it up into tiles for serving, saving considerable work at runtime. Angus Dorbie later wrote an entire custom tool for us to make this even more efficient. David Kornmann added a continuous level of detail (CLOD) 3D terrain system that handled arbitrary surface detail.
If you look at their original videos, TerraVision shows popping and visible seams between tiles when they change resolution, moving closer or farther from any spot of land. This looks more like the way both Microsoft and NASA implemented their “Virtual Earth” and “WorldWind” apps. It may take expert eyes to notice, but Google Earth is truly seamless. You can fly anywhere on Earth in about 2.5 seconds with no lag and only blurriness if the tiles don’t arrive fast enough. It was able to run on modest 1999 PCs that only had 56k modems for network and 16–32MB of GPU texture memory.
ART+COM always required a very expensive SGI supercomputer, AFAIK.
Showing two planet browsers side-by-side, as the lawyers are depicted in the TV show, doesn’t mean one of them must be stolen from the other. All 3D planet browsers are supposed to look like this, smoothly roaming and zooming. In fact, I even added cloud layers in my first UX prototypes, as well as stars, sun and moon.
If you’re looking for inspiration, we could cite examples from the 1970s to the early 90s, well before ART+COM did any work on this. It might be the same public inspiration for all of us (e.g., “Powers of Ten”). I’d certainly been dreaming of this since the early 90s at least.
We added many innovations that ART+COM didn’t even think about. For example, map makers spend weeks or months designing the labels for places and roads on maps. There’s both an art and a science to it. I learned some of their techniques. But our real-time system had to draw all of the labels in just 2–3 milliseconds, sixty times per second.
I also worked hard to make the UI more fun and responsive, using a common computer mouse to fly around — no giant trackball required. It had to feel right, responsive but also conveying a sense of mass for a real planet. I also invented a system that’s similar today to Functional Reactive Programming to allow us to reprogram the guts of the app based on a simple declarative data-flow syntax. That made Keyhole’s EarthViewer fast and extensible. Early work here eventually lead to a global standard geospatial file format called KML.
These are just a few of the ways Keyhole went beyond anything I’ve seen from ART+COM, and way before it had any resources from Google. Yet my startup was entirely left out of this dramatization arguing inventorship.
So What Did We Learn?
If Google had stolen anything from ART+COM, I would not defend it. I’m openly critical of their ad-driven business model, especially since the damage is now much clearer. It’s one of the main reasons I sold all my Google stock after the acquisition. I respect Google’s engineering work and the great “free” information they provide. There are a lot of smart and dedicated people working there. But the ad-tech biz model is not ok, IMO.
That issue is broached briefly in the show, but only to point out how much money Google would theoretically owe these guys if they licensed the tech at the exceptional rate of 10 cents per use — hence Billions on the table. Such a deal was impossible to obtain, which may be one reason talks stalled. Google allegedly offered ART-COM some modest money to make the dispute go away. It was evidently worth a few million to prevent such an expensive lawsuit, but not billions.
My personal goal for Keyhole was always to offer a highly efficient “3D trellis” of useful spatial data for anyone to add their own data on top of (while retaining their individual legal rights to their data). This would have been the basis for our future in Augmented Reality. Google has barely scratched the surface of this virtual globe and its potential benefits to society, even today. There’s much more work to do.
Yet for what it was, all of the inventors I’ve mentioned by their real or fictional names deserve credit for their pioneering work here. All of them. Google may have acted like a billion-pound gorilla towards multiple small companies in recent years. And that could be relevant for future anti-trust sanctions. However, I don’t believe anything was ever actually stolen from, copied from, or even originally inspired by ART+COM.
Whenever a movie says “based on a true story,” it’s likely to be dramatizing and fictionalizing large chunks of the story, for dramatic, political, or other purposes. The filmmakers admitted publicly to not even checking in with some of people who might disagree with their single-minded version of events, because: why bother?
“We never approached them. They would have never jumped on board anyway, because these big tech companies want to control and protect their company’s image. We just took the risk, deciding to name them, and now we will see. We don’t know how they will react or if they will sue us.”
That an incredible admission of “documentary negligence” and a stunning lack of curiosity in favor of personal bias. They told their story, period.
You can interpret the real story as being about greedy companies trying not to pay inventors what they owe. Or, perhaps a greedy law firm seeking a $300M dollar return on a $10M investment. Or naive artists and makers who don’t know how to protect their work or make effective business deals. Or perhaps it’s a story, indirectly, about blithe filmmakers on a mission, who didn’t care about the actual truth of the matter.
You can also recognize that even if the dialog in the show was 100% factual (and it strains credulity) there is a much simpler explanation to the story than vast CIA conspiracies and totally unhelpful IP theft.
ART+COM’s patent was invalidated in 2017 because another group, the Sarnoff Research Center (SRI) in Palo Alto had publicly shown a similar system a full year earlier. Like ART-COM’s demo, it was also called “TerraVision”, and it constituted prior art. They didn’t simply lose their infringement case against Google, but got their entire IP claim obliterated.
In supreme irony, the people asserting they “invented” Google Earth first were finally undone by an even earlier system with the same name and functionality as theirs.
In my view, a bunch of smart people were separately trying to solve similar problems around the same time. They used their wits and the tools in the room to do so. Let’s celebrate their ingenuity and help everyone to achieve more in the future (and hopefully without the exploitive business models).
UPDATE (October, 2022): If you’re still in doubt or don’t trust me for whatever reasons, consider that ART+COM’s demo as shown in the Netflix drama relied on these same expensive high-end SGI computers and likely the “hardware clip-mapping” feature that made it possible to render a very large “whole-earth” virtual texture.
Does that mean that SGI invented the hardware first, then ART+COM was able to use it to do something cool. And then people from SGI somehow copied ART-COM ten or twenty years later?
Think about who owned the spy satellites that fed the data one would use to ship a whole-earth detailed map. It wasn’t until a few years later when these aerial imagery datasets were finally declassified and made available for a small company like Keyhole to re-publish (hint: Keyhole was named after the original KH spy satellites and got some funding from the government).
Is it at least possible that all of this impressive tech was first developed primarily for the US government and military, and that people who are properly disclosed on that probably still can’t talk about it?
There is certainly no need for vast conspiracy theories when the truth is that simple. ART+COM never had the facts on their side to win against Google, SGI, Microsoft, or anyone else they felt slighted by.
