Taha Erdem Ozturk

Columbia GSAPP
Schermerhorn Hall Ext #654
1172 Amsterdam Avenue
New York, NY 10027


Home
About
Contact
Center for Spatial Research





COLUMBIA UNIVERSITY  —  CENTER FOR SPATIAL RESEARCH
FALL 2023



DEVELOPER BLOG

KEEPING TRACK OF THE WORK


Using Zero-Shot Depth Estimation Models on StreetView images
apr 18, 2024

Entry to be logged very soon.


Appending ML-Generated Data with GIS Datasets Using File Indexing System
apr 18, 2024

Entry to be logged very soon.



Using GroundingDINO + Segment Anything for Semantically Processing Facades
apr 18, 2024

Entry to be logged very soon.



Creating an Indexing System
apr 18, 2024

Entry to be logged very soon.



Underlaying StreetView Availability Layer in QGIS
apr 18, 2024



To underlay Google’s own StreetView Availability graphic (the blue lines showing the roads Google has StreetView images for, displayed right before releasing the pegman down on the map) all you have to do is to add it just as another Google Maps Raster Tileset. 

  1. In the Browser panel, right-click XYZ Tiles > New Connection

  2. Enter a name like "Google StreetView Coverage" and the URL: https://mts2.google.com/mapslt?lyrs=svv&x={x}&y={y}&z={z}&w=256&h=256&hl=en&style=40,18

  3. Double-click the new XYZ Tile to add it to the map

There you go, all set!



Mapping Adjacent Buildings
Nov 13, 2023

In order to minimize the amount of Google StreetView requests, it was crucial to find out if two buildings in QGIS had adjacent walls, so that we can locate these intersecting points, which will be useful to request static images only from (and looking towards) these points.
  

Step 1: Identify Adjacent Buildings


  1. Buffer the Buildings: Create a small buffer around each building to ensure that touching walls are detected. Go to Vector > Geoprocessing Tools > Buffer. Set a small buffer distance (like 0.5 meters) to account for minor inaccuracies in the building outlines.

  2. Select Adjacent Buildings: Use the   Select by  Location  tool (Vector > Research Tools > Select by Location) to select buildings that intersect the buffered layer. This will identify buildings that are adjacent.

Step 2: Create Points on Joint Edges


  1. Intersection of Buffers: Use the  Intersection   tool (Vector > Geoprocessing Tools > Intersection) on the buffer layer to find intersections between buffers of adjacent buildings. This will give you the lines where the buildings are adjacent.

  2. Convert Intersection Lines to Points: Use the   Extract vertices  tool (Vector > Geometry Tools > Extract vertices) on the intersection lines. This will create points at the vertices of the intersection lines, which includes the points where the buildings' walls meet.

Step 3: Save or Export the Refined Layer as a CSV Table


After refining the intersection results, save or export the layer for further use. Additionally, I’ve exported these points in LAT and LON as a CSV layer to further processing. Ensure that your layer is in a geographic coordinate system (Google Maps primarily uses the Web Mercator projection, or EPSG:3857) that uses latitude and longitude. If not, you may need to reproject your layer to such a coordinate system.

Follow these steps to export LAT and LON values of my intersection points as a CSV table:

  1. Open Attribute Table: Right-click on the layer and select 'Open Attribute Table'.

  2. Add Latitude and Longitude Columns: Use the Field Calculator to create new fields for latitude and longitude. Use expressions like   y($geometry)  for latitude and     x($geometry)  for longitude. Make sure to set the output field type to 'Decimal number'.

  3. Export Data: Once the new fields are populated, you can export this data. Right-click on the layer, select 'Export', and then 'Save Features As'. Choose 'CSV' as the format to easily use the data in other applications.


Step 4: Querying Google StreetView API


  1. Google Street View API Key: You'll need an API key to use the Google Street View API. You can get this from the Google Cloud Console.

  2. Prepare Your Query: Using the exported CSV file, you can prepare your queries for the Google Street View API. Each query will require the latitude and longitude values. You might use a programming language like Python to automate this process.

  3. API Query Structure: A typical Street View API query looks like this:
    <https://maps.googleapis.com/maps/api/streetview/metadata?location=LAT,LON&key=YOUR_API_KEY>
    Replace LAT and LON with your latitude and longitude values, and YOUR_API_KEY with your actual API key.

  4. Extract panoIDs: The response from the Street View API will include metadata about the nearest Street View image, including the panoID. You can parse this response to extract the panoID.

  5. Handle API Limits and Quotas: Google StreetView API has

  6. Storing and Using panoIDs: Once you have the panoIDs, you can store them for further use or analysis. Depending on your end goal, you might use these IDs to retrieve specific Street View images or other related tasks.

This code below does the steps below:
  1. Fetch a session token.
  2. Get panorama IDs for given locations.
  3. Fetch latitude and longitude for each panorama ID.
  4. Process locations from a CSV file in chunks, fetching panoIDs and their corresponding latitude and longitude.
  5. Write the results to a new CSV file.
import requests import pandas as pd# Function to get a session token def get_session_token(api_key): url = "<https://tile.googleapis.com/v1/createSession?key=>" + api_key headers = {'Content-Type': 'application/json'} payload = { "mapType": "streetview", "language": "en-US", "region": "US" } response = requests.post(url, json=payload, headers=headers) if response.status_code == 200: return response.json().get('session') else: print("Error getting session token:", response.status_code, response.text) return None# Function to get a single panorama ID def get_panorama_id(session_token, api_key, location): url = f"<https://tile.googleapis.com/v1/streetview/panoIds?session={session_token}&key={api_key}>" headers = {'Content-Type': 'application/json'} payload = {"locations": [location], "radius": 50} response = requests.post(url, json=payload, headers=headers) if response.status_code == 200 and 'panoIds' in response.json(): pano_ids = response.json()['panoIds'] return pano_ids[0] if pano_ids else None else: return None# Function to get latitude and longitude for a given panorama ID def get_pano_lat_lon(api_key, pano_id): url = f"<https://maps.googleapis.com/maps/api/streetview/metadata?pano={pano_id}&key={api_key}>" response = requests.get(url) if response.status_code == 200 and 'location' in response.json(): data = response.json() return data['location']['lat'], data['location']['lng'] else: return None, None# Function to process a chunk and fetch panoIDs and their lat-lon def process_locations_and_fetch_pano_ids(session_token, api_key, chunk): chunk['panoID'] = None chunk['panoLat'] = None chunk['panoLon'] = None for i, row in chunk.iterrows(): location = {"lat": row["LAT"], "lng": row["LON"]} pano_id = get_panorama_id(session_token, api_key, location) chunk.at[i, 'panoID'] = pano_id if pano_id: pano_lat, pano_lon = get_pano_lat_lon(api_key, pano_id) chunk.at[i, 'panoLat'] = pano_lat chunk.at[i, 'panoLon'] = pano_lon return chunk# Main execution api_key = 'YOUR_API_KEY' # Replace with your actual API key csv_file = 'sirinevler.csv' output_csv_file = 'sirinevler_with_panoID_and_LatLon.csv'session_token = get_session_token(api_key) if session_token: all_chunks = [] for chunk in pd.read_csv(csv_file, chunksize=100): updated_chunk = process_locations_and_fetch_pano_ids(session_token, api_key, chunk) all_chunks.append(updated_chunk) # Concatenate all chunks and write to a new CSV file updated_data = pd.concat(all_chunks) updated_data.to_csv(output_csv_file, index=False)

This code allows us the get LAT and LON values of the closest panoID location from a given intersection point. Now, we can calculate the viewing angle from the panoID location towards the intersection point in QGIS. This way we can add the looking angle when querying the imagery.

Step 5: Calculating the Camera Angle


Calculating the angle at which a given point in your location layer is viewed from the closest point in your panorama ID (panoID) location layer in QGIS involves a few steps. You'll need to first find the closest panoID point to each point in your location layer and then calculate the bearing (angle) from the panoID point to your location point.

Here's a step-by-step guide to achieve this:

Step 1: Find the Closest PanoID Point for Each Location


  1. Open the QGIS Field Calculator:
    • Select the layer that contains your locations.
    • Open the attribute table and click on the "Open Field Calculator" icon.
  2. Create a New Field:
    • Check the "Create a new field" option.
    • Name the field (e.g., closest_panoID).
  3. Use an Expression to Find Closest PanoID:
    • In the expression box, use a function to find the closest feature in the panoID layer. QGIS doesn't have a direct function for this, but you can use a combination of functions like aggregate to calculate the minimum distance and then find the corresponding panoID. This can be complex and might require custom scripting.


Step 2: Calculate the Heading Values


After finding the closest panoID for each location, you can calculate the bearing. This also requires using the Field Calculator:

  1. Open the Field Calculator Again:
    • Still in the attribute table for your location layer, open the Field Calculator.

  2. Create a New Field for Bearing:
    • Check "Create a new field."
    • Name the field (e.g.,  bearing .)

  3. Use an Expression to Calculate the Bearing:
    • Use a QGIS expression to calculate the bearing. The basic formula for bearing between two points (lat1, lon1 and lat2, lon2) is as follows:

    • degrees(atan2(sin(lon2-lon1)*cos(lat2), cos(lat1)*sin(lat2)-sin(lat1)*cos(lat2)*cos(lon2-lon1)))
    • You'll need to replace  lat1 ,  lon1 ,  lat1 ,  lon2 with the respective field names of your datasets.

To calculate the camera heading (angle) for each point in your dataset relative to its matched panoID location in QGIS, you'll need to compute the bearing from each panorama point to its corresponding reference point. The bearing will be the heading angle from North in degrees, ranging from 0 to 360.

Here's how to calculate the heading using the Field Calculator in QGIS:

  1. Open Your Layer in QGIS:
    • Load your layer that contains both the reference points and their matched  panoLat  and  panoLon  values.

  2. Open the Field Calculator:
    • Open the attribute table for your layer.
    • Click on the "Open Field Calculator" icon.

  3. Create a New Field for Heading:
    • Check the option "Create a new field".
    • Name the field (e.g.,  heading ).
    • Set the field type to Decimal number (real).

  4. Calculate the Heading:
    • In the expression box, use the following expression to calculate the bearing:
      degrees(atan2(sin(radians("panoLon" - "longitude")) * cos(radians("panoLat")), cos(radians("latitude")) * sin(radians("panoLat")) - sin(radians("latitude")) * cos(radians("panoLat")) * cos(radians("panoLon" - "longitude"))))
    • Here, replace " latitute " and " longitude " with the field names of your original point's coordinates, and " panoLat " and " panoLon " with the field names of your panorama point's coordinates.

  5. Adjust the Result to 0-360 Degrees Range:
    • The above formula might return negative values. To adjust all bearings to a 0-360 range, which is what Google StreetView API uses, you can modify the expression:
      (degrees(atan2(sin(radians("panoLon" - "longitude")) * cos(radians("panoLat")), cos(radians("latitude")) * sin(radians("panoLat")) - sin(radians("latitude")) * cos(radians("panoLat")) * cos(radians("panoLon" - "longitude")))) + 360) % 360
  6. Run the Field Calculator:
    • Click "OK" to add the  heading values to your layer.

  7. Save the Changes:
    • If your layer is editable, save these changes.

This procedure computes the heading angle for each point relative to its matched panorama location. The result is a heading value that indicates the compass direction from each panorama point towards its corresponding reference point, following the standard compass degrees.

Step 3: Querying the Images


Now, by exporting these points into a CSV file, we can once again request and save StreetView static image for every item on this list, by using 'index' column when saving the images. Each item has an index #, and a panoID. Each image also has a heading value for adjusting the viewing angle. To be able to as much of the buildings’ height as possible, and since we’re only interested in obtaining the adjoint parts of these buildings - as that is enough to make floor slab alignment predictions, we set our resolution to 200x800.

Then we modify our script to query all Static image tiles from panoID locations.

  1. Read the CSV File:
    • Use the  pandas library to read your CSV file and iterate through each row.

  2. Generate API Request URL:
    • For each row, construct the Google Street View API URL using the  panoId and  heading values.

  3. Request and Save the Image:
    • Use the  requests library to fetch the image from the constructed URL.
    • Save the image to a file, using the  index column value for the filename.
import pandas as pd import requestsdef save_street_view_images(csv_file, api_key): # Read the CSV file df = pd.read_csv(csv_file) for index, row in df.iterrows(): # Construct the API URL url = f"<https://maps.googleapis.com/maps/api/streetview?size=200x800&pano={row['panoID']}&heading={row['heading']}&key={api_key}>" # Request the image response = requests.get(url) # Check if the request was successful if response.status_code == 200: # Save the image with open(f"streetview_{row['index']}.jpg", 'wb') as file: file.write(response.content) else: print(f"Failed to fetch image for index {row['index']}")# Usage api_key = 'YOUR_API_KEY' # Replace with your actual Google API key csv_file_path = '/path/to/your/Sirinevler - Final.csv' # Replace with the path to your CSV file save_street_view_images(csv_file_path, api_key)

The images are saved into our folder, each with their respective index numbers — which will come in handy when combining them back into our map.



How to Obtain All
Panoramic Imagery on A Geo-Path?

Nov 10, 2023

For this part of the project, this Stackoverflow entry was very helpful. I built a new Python project, and installed requests libraries to work with JSON queries we need to request using Map Tiles API and StreetView Static API. In order to use Map Tiles API, you also need to get a session token. This code allows us to first obtain a session token through our Google Maps API Key; and then use that session token to make Map Tile queries, through which we obtain closest StreetView  panoID s of any given Latitude and Longitude value. 

To keep it simple, I first used two points - start and end points of a single city block in the same neighborhood:
Start of the Street: 40.9969411,28.8351982

End of the Street: 40.9967711,28.8383787

Used these values as part of our query:
import requests

# Function to get a session token
def get_session_token(api_key):
    url = "https://tile.googleapis.com/v1/createSession?key=" + api_key
    headers = {'Content-Type': 'application/json'}
    payload = {
        "mapType": "streetview",
        "language": "en-US",
        "region": "US"
    }

    response = requests.post(url, json=payload, headers=headers)
    if response.status_code == 200:
        return response.json().get('session')
    else:
        print("Error getting session token:", response.status_code, response.text)
        return None

# Function to get panorama IDs
def get_panorama_ids(session_token, api_key):

    url = f"https://tile.googleapis.com/v1/streetview/panoIds?session={session_token}&key={api_key}"
    headers = {'Content-Type': 'application/json'}
    payload = {
       "locations": [
            {"lat": 40.9969411, "lng": 28.8351982},
            {"lat": 40.9967711, "lng": 28.8383787},
          ],
          "radius": 50
    }

    response = requests.post(url, json=payload, headers=headers)
    if response.status_code == 200:
        print(response.json())
    else:
        print("Error getting panorama IDs:", response.status_code, response.text)

# Main execution
api_key = ‘YOUR’_API_KEY
session_token = get_session_token(api_key)
if session_token:
    get_panorama_ids(session_token, api_key)

The query response:
'panoIds': ['sKZAWeO4Tvunma3aQPo-xA', 'pH1sPcms3iyNvzpYNNXv4g']

To get the maximum possible existing imagery from the same block, we divided path into 1m intervals, which resulted in 12 points. Hence, the updated code had the locations for these Latitude and Longitude values:
{"lat": 40.9969280, "lng": 28.8354429}
{"lat": 40.9969149, "lng": 28.8356875}
{"lat": 40.9969019, "lng": 28.8359322}
{"lat": 40.9968888, "lng": 28.8361768}
{"lat": 40.9968757, "lng": 28.8364215}
{"lat": 40.9968626, "lng": 28.8366661}
{"lat": 40.9968496, "lng": 28.8369108}
{"lat": 40.9968365, "lng": 28.8371554}
{"lat": 40.9968234, "lng": 28.8374001}
{"lat": 40.9968103, "lng": 28.8376447}
{"lat": 40.9967973, "lng": 28.8378894}
{"lat": 40.9967842, "lng": 28.8381340}

The query response:
'panoIds': ['sKZAWeO4Tvunma3aQPo-xA', 'xVx18pNmKYGOw5tj1QgZ8g', '8rOI3B4xCN3R4m9ZG4L3Uw', 'rw9w9zo691AM0PHhT1fGkw', 'wkk6IeakUMDyVpCO-ACp1A', 'I_4W9XFbThVK0DdmlJXEjw', 'pl_wqCCcV4YE1-Q_YCA_Dg', '8yy1ol55N39GMWYrYdXmCg', 'cxT2FSDTA6WqkUwPX7U03g', 'uY3u1gK4sDM4Ivijl8VkIw', 'dO1JokjpgGxoa5NZbL9MVA', 'pH1sPcms3iyNvzpYNNXv4g']

Knowing and having been able to automate this process, now we can obtain all panoramic imagery for all points that has a StreetView Panorama View along the path. For this, I just modified my previous Python script with a loop, which allowed me to download all required imagery from each  panoID  location.
from streetview import get_streetview

# List of panorama IDs
pano_ids = ['sKZAWeO4Tvunma3aQPo-xA', 'xVx18pNmKYGOw5tj1QgZ8g', '8rOI3B4xCN3R4m9ZG4L3Uw', 'rw9w9zo691AM0PHhT1fGkw', 'wkk6IeakUMDyVpCO-ACp1A', 'I_4W9XFbThVK0DdmlJXEjw', 'pl_wqCCcV4YE1-Q_YCA_Dg', '8yy1ol55N39GMWYrYdXmCg', 'cxT2FSDTA6WqkUwPX7U03g', 'uY3u1gK4sDM4Ivijl8VkIw', 'dO1JokjpgGxoa5NZbL9MVA', 'pH1sPcms3iyNvzpYNNXv4g']
api_key = "YOUR_API_KEY"

for pano_id in pano_ids:
     image = get_streetview(pano_id=pano_id, api_key=api_key)
    image.save(f"{pano_id}.jpeg", "jpeg")




Implementing the
Google StreetView API
Nov 8, 2023


Getting started with the example Google StreetView Static API Code, derived from Google Maps API Documentation:
https://maps.googleapis.com/maps/api/streetview?size=600x300&location=46.414382,10.013988&heading=151.78&pitch=-0.76&key=YOUR_API_KEY&signature=YOUR_SIGNATURE

Pilot neighborhood location:
Kurultay Sk. Sirinevler, Bahcesehir/ISTANBUL

LAT/LON: 40.9969122,28.8355952

Modified Google StreetView Static API Code:
https://maps.googleapis.com/maps/api/streetview?size=600x300&location=40.9969122,28.8355952&heading=151.78&pitch=-0.76&key=API_KEY_HERE

The resulting image. It works, but the query heading, pitch, zoom, and aspect ratio needs to be adjusted. 
Useful paramaters for API requests:
  •  heading indicates the compass heading of the camera. Accepted values are from 0 to 360 (both values indicating North, with 90 indicating East, and 180 South). If you don't specify a heading, a value is calculated that directs the camera towards the specified   location , from the point at which the closest photograph was taken.

  •  fov (default is 90) determines the horizontal field of view of the image expressed in degrees, with a maximum allowed value of 120. When dealing with a fixed-size viewport, as with a Street View image of a set size, field of view in essence represents zoom, with smaller numbers indicating a higher level of zoom.

  •  pitch (default is 0) specifies the up or down angle of the camera relative to the Street View vehicle. This is often, but not always, flat horizontal. Positive values angle the camera up (with 90 degrees indicating straight up); negative values angle the camera down (with 90 indicating straight down).

  •  radius (default is 50) sets a radius, specified in meters, in which to search for a panorama, centered on the given latitude and longitude. Valid values are non-negative integers.

The next step was to play with the Static API variables to see whether it could be useful for generating front-facing facade imagery of each side of the street.

1. FOV and Pitch Corrected
https://maps.googleapis.com/maps/api/streetview?size=600x300&location=40.9969122,28.8355952&heading=151.78&pitch=0&fov=120&key=API_KEY_HERE

This allowed for a more zoomed-out view of the same shot. 

2. Adding  pano  in the request.
https://maps.googleapis.com/maps/api/streetview?size=600x300&location=40.9969122,28.8355952&pano=&key=API_KEY_HERE

This didn’t return a panoramic cylindrical projection, yet interestingly adjusted the camera angle to 180/0 degrees for a front-facing shot. 
3. Playing with the aspect ratio.
https://maps.googleapis.com/maps/api/streetview?size=640x640&location=40.9969122,28.8355952&pano=&key=API_KEY_HERE

Google StreetView Static API only returns up to 640x640 pixels of image tiles for each query. 
4. Trying out a very skinny aspect ratio to capture full length of the building. 
https://maps.googleapis.com/maps/api/streetview?size=200x800&location=40.9969122,28.8355952&pano=&key=API_KEY_HERE

This certainly worked, however the quality gets increasinly low towards the top. Another challenge is the aspect ratio is very narrow for even for a mid-size residential building to be taken out of the shot without being cropped. 

This may become a challenge, as many streets do not have dense enough panorama shots that would allow us to stitch missing imagery to generate a complete front-facing image of the facade.

Moving on, we also wanted to try to get a full panoramic image of the city, to see whether we could obtain higher-resolution image tiles out of the panorama. To do this, first we need the  panoID  metadata, which we can obtain by adding a metadata? parameter into our request. 

https://maps.googleapis.com/maps/api/streetview/metadata?parameters

https://maps.googleapis.com/maps/api/streetview/metadata?size=640x640&location=40.9969122,28.8355952&key=API_KEY_HERE

Response:
{
  "copyright" : "© Google",
  "date" : "2022-06",
  "location" :
    {
        "lat" : 40.99692394020791,
        "lng" : 28.83559648841946
    },
  "pano_id" : "bRKZXUcA_NYAjOohhEltxA",
  "status" : "OK"
}

Success! By using the pano_id we requested, we can now obtain a full panoramic cylindrical projection view of the street, using a Python library called “streetview”.

First step was to create a new Python project, and installing the required library within our virtual environment:
pip install streetview

Then on main.py, let’s create a simple Python app that saves the panorama image into our project folder as “image.jpg”:
from streetview import get_panorama

image = get_panorama(pano_id="bRKZXUcA_NYAjOohhEltxA")

image.save("image.jpg", "jpeg")

Resulting image was promising, although now the challenge was to defisheye the panoramic projection:


Going one step back, since this image was even more challenging to be used for ML data-training, maybe getting two sides of the street imagery by using StreetView Tile images could be more useful. On the contrary to the skinny aspect ratio query we’ve been able to obtain earlier, these Tiles are higher in resolution, which may help with the quality loss on the skinny aspect-ratio image we’ve ended up with earlier. 

For this, I created a new Python project and installed  requests  and   PIL  libraries, which allows us to work with JSON requests and processing images respectively. Then requested 640x640px static tiles for our panoID location.
import requests
from PIL import Image
from io import BytesIO

# Replace with your Google API key
API_KEY = 'YOUR_API_KEY'

def download_tile(pano_id, x, y, zoom):
    # Construct API URL for a specific tile
    url = f'https://maps.googleapis.com/maps/api/streetview?pano={pano_id}&size=640X640&fov=90&heading={x}&pitch={y}&key={API_KEY}' 

    response = requests.get(url)
    if response.status_code == 200:
        image = Image.open(BytesIO(response.content))
        image.save(f'{pano_id}_{x}_{y}.jpg')
    else: print(f"Error fetching tile {x}, {y}")

pano_id = 'bRKZXUcA_NYAjOohhEltxA'
zoom = 1 # Zoom level
for x in range(0, 360, 90): # Adjust range and step based on zoom and desired overlap
    for y in range(-90, 91, 90): # Adjust range and step based on zoom
        download_tile(pano_id, x, y, zoom)

Resulting sequences of images:


By changing the angle values, we can limit our query down to a useful batch:
180_-90
180_-45
180_0
180_45
180_90

---

0_-90
0_-45
0_0
0_45
0_90

Adjusting the code with these values:
import requests
from PIL import Image
from io import BytesIO

# Replace with your Google API key
API_KEY = 'YOUR_API_KEY'

def download_tile(pano_id, x, y, zoom):
# Construct API URL for a specific tile
    url = f'https://maps.googleapis.com/maps/api/streetview?pano={pano_id}&size=640X640&fov=90&heading={x}&pitch={y}&key={API_KEY}' 

    response = requests.get(url)
    if response.status_code == 200:
        image = Image.open(BytesIO(response.content))
        image.save(f'{pano_id}_{x}_{y}.jpg')
    else: print(f"Error fetching tile {x}, {y}")

pano_id = 'bRKZXUcA_NYAjOohhEltxA'
zoom = 1 # Zoom level
for x in range(0, 360, 180): # Adjust range and step based on zoom and desired overlap
    for y in range(-90, 91, 22): # Adjust range and step based on zoom
        download_tile(pano_id, x, y, zoom)

Voila! Now we have both sides of the street, with higher-res 640x640px imagery:
Left side

Right side

Stitching both image sequences using Photoshop Batch Image Processing:
Left side

Right side