Urban Jungle Street View – Behind the scenes

Introduction

Urban Jungle Street View is an experiment that I did to play with the depth data in Google Steet View. I knew that the information was there but had not successfully managed to parse the information. But some weeks ago I found a library that provided this for me. About the same time I started to play “Last Of Us” from Naughty Dogs. About 15 minutes in, stunned by the environments, I just had to try out this idea instead (damn you inspiration).

last-of-us

Here is a screenshot with the final result: endresult

I posted the link on Twitter and the response was overwhelming. About 200.000 visits the first week and it reached the front page on theverge.com, dailymail.co.uk, wired.com, psfk.com among others. Even television picked it up, got interviewed by the Japanese TV show “Sukkiri!”. Something triggers the inner adventurer in the audience for sure 🙂

2D Map

map2d Using the Google Maps API is so easy and powerful. So much data in your hands. For the style of the map, I use the Styled Maps Wizard to see what different kind of settings there are and export the settings as JSON. One tips is to turn off all features first with

{"stylers": [
    { "visibility": "off" },
]},

and after that turn on the things you want to show and style.

Street view coverage layer

I use the StreetViewCoverageLayer while dragging the little guy to show all available roads in Street View. It’s not just roads but also indoor places and special sites like Taj Mahal. I like the effect in Google Maps when holding Pegman over a road and he is snapped to the road with a direction-arrow aligning the road. But that functionality is currently only available in the Google Maps website and not in the API version 3. But I made a custom version that works not near as good but close enough. You can calculate the pixel-position and the tile-index and then load the coverage-tile manually, write it to a canvas and read the pixel at the corresponding local position. If the pixel is not transparent then we hover a street-view enabled road or place. I also inspect the color to avoid changing the state if you hover a indoor-panorama, which is orange. You can still drop him, but it adds a visual queue that it’s not optimal to drop him there. Sometimes you end up inside a building anyway, but I have not found a way to notice that without loading the data first.

Street View

I use a library called GSVPano, created by Jaume Sanchez, to show the panoramas in WebGL and three.js. The source is removed from the repo though. To access the depth and normal data I used a different library. Read more about that here. depthmap

The data for each panorama is not a depth-map directly (from the API) but instead stored in a Base64-encoded string to save bandwidth, requests and in a way that makes it quick to parse for the actual purpose of it; to find the normal (direction of a surface) and the distance on a point in the panorama to aid the navigation UI. If you hover the ground with the mouse-cursor in Google Streetview you get the circle cursor and if you point at a building you get a rectangle aligned with the surface of the building. I’m using that behaviour as well when plotting out the foliage, but also passing the data to the shader as textures.

In the demo, the normal-map takes care of making the ground transparent in the shader, so I can place a plane under the panorama-sphere. The ground plane will be visible through the transparent ‘hole’ (could maybe create geometry instead somehow). The depth-map is used to make a interpolation from transparent to opaque ground. The depth-data has a limited range (the visible range), so it will look strange otherwise with a hard edge.

I can also add a little fog to the environment. The planes in the map is not exactly matching the real-world geometry so it can not be too much fog, because the edges will be visible around the buildings. In the end I removed it almost completely since I liked a more sunny weather, but here is screenshot with some more added to it:

extra-fog

Foliage

Foliage are sprites (flat billboards) or simple geometry that is positioned in 3d space. To place them I look in random positions in a predefined area of the texture. First, to get the sense of how a flat image is wrapped around a sphere, look at this image: uv-wrap Image from Wikipedia

Here you see the depth-data parsed and displayed as a normal texture. The dots is where I calculated things to grow (I describe that more below).

normalmap

When selecting a point in the texture you get the position on a sphere like this:

var lat = u * 180-90;
var lon = v * 360-180;
var r = Math.cos(DEG_TO_RAD * lat);

//range between 0-1
var pos = new THREE.Vector3();
pos.x = (r * Math.cos(DEG_TO_RAD * lon) );
pos.y = (Math.sin(DEG_TO_RAD * lat));
pos.z = (r * Math.sin(DEG_TO_RAD * lon));

But if all objects are placed at a fixed distance they will the same scale, so the distance from the camera is important, like pushing the object away from the camera in the same angle. That value is stored in the depth data so:

pos.multiplyScalar(pointData.distance);

If I want the object to point at the camera, like the trees, it’s just to set

myPlantObject.lookAt(camera).

But If the object should align the surface it suppose to stick to, we should use the normal value, the direction the wall in facing:

var v = pos.clone();
v.add( pointData.normal );
myPlantObject.lookAt(v);

normalmap3d

Here is a demo with this setting. To get a true 3d perspective, the sprite could be replaced with real geometry, but leafs are quite forgiving to perspective issues and sprites are much more performant. If the angle to the camera is too big, it looks really strange though.

If you click or tap the scene you can put more foliage into it. Then it’s just the same process but reversed. Get the collision point on the sphere, convert it to UV-space (0-1) on the texture. Test against the data and get the preferred plant for that position if any.

Fine tuning

Every time you visit a place it’s randomly generated. There is some estimations that I use to get a bit more smarter distribution of plants. If the calculated direction is pointing upwards, make grass growing on the ground. If it’s perpendicular to the ground, we can put some wall foliage sprites there. To hide the seam between the ground and the walls I do a trick to find that edge. The ground has always the same normal, as you can see as the pink color above. I iterate the pixels in the texture from bottom and up to find the first pixel in a different color than the ground. That is where the edge is. Put a grass-sprite there and go to the next column of pixels to do the same. To enhance the feeling of walls even further lets introduce another type of plant: vines. These have stalks made of 3d geometry (a modified ExtrudeGeometry extruded along a bezier path, or “spline”) with sprite-based leafs. To detect a good place for them to grow I make another lookup in the normal-map from where I detected the edge previously. About 50 pixels up from these points I test if the value is still the same as the wall I found first. If it’s a different color it’s probably a small wall and not an optimal place for a vine to grow, otherwise I can place it there but not too many so test for a minimum distance between them. It’s buggy sometimes, but it’s close enough for this demo. A couple of trees is also inserted as sprites pointing toward the camera. They could be more procedural for a more interesting result.

Navigation

The link to nearby panoramas is included in the API-result so it’s just to add the arrows and detect interactions with THREE.RayCaster (example). The effect between the locations is part of the post effects pass, I just set the amount of blur in a blur pass…

Post effects

Again, Jaume Sanchez has more valuable source code in form of Wagner, a post-effect composer for three.js. To make the panorama layer and the 3D layer blend together a couple of effects are added to the end result. First a Bloom-filter, which adds glow to bright areas. And a dirt-pass, that adds the dust and raindrops to the composition. The final touch is a little movement to camera so the scene look more alive and less of a still image. There is definitely room for improvement, especially with the color balance in the scene, and more advanced geometry and procedurally generated plants.

Try it now

Now when that is out of the system, lets return to “Last Of Us” finally…