Why I Created GeoDownloader.com

Working with OpenStreetMap data is incredibly rewarding for geospatial projects, but it can also be frustrating. The data is rich and comprehensive, but accessing it in a way that fits your project’s needs can be time-consuming. I’ve personally spent hours wrestling with overly complex workflows, converting file formats, and extracting specific data from larger datasets. While there are websites that help you download data, I don’t find them intuitive. Thus, I decided to build another one to make it easier for myself and anyone else who faces similar challenges.

Downloading data for a specific area with a small number of features can be particularly time-consuming. There are several options for downloading data through GIS applications like QGIS, but I wanted to create something that runs in your browser for small tasks (for now 😉).

That’s how GeoDownloader.com was born – a website where you can quickly download exactly the data you need, in the format you need, without any unnecessary hassle.

How to use

I’ve designed GeoDownloader with a simple and approachable interface, making it easy for users new to GIS or OpenStreetMap to start downloading data immediately.

The intuitive map interface allows you to visually select your area of interest (AOI). Simply draw a shape on the map, and GeoDownloader will automatically capture all features within that boundary.

You can easily customize your download by deselecting any unwanted features from the list, ensuring you only get the data you need.

Understanding the importance of compatibility in GIS workflows, GeoDownloader supports exports in three formats:

• GeoPackage

• Shapefile

• GeoJSON

Choose the format that best suits your tools, whether you’re using QGIS, ArcGIS, or other software. Let me know in the comments if you need additional format options.

Filtering Tools

One of my favorite features is the ability to filter selected data. You can refine your selection based on:

• Tags: Select features by OSM tags like highway, building, or landuse.

• Tag Values: Narrow down by specific tag values, such as highway=primary or building=school.

• Geometry Type: Focus on specific geometry types, like points, lines, or polygons.

These filters give you precise control over what you’re downloading, saving you the effort of cleaning or preprocessing data later.

Limits

There are some important limitations I need to explain. I created this tool to simplify downloading small datasets from OpenStreetMap, but there are certain constraints. Instead of proxying third-party APIs like Overpass API, I host all OSM data on my server in an indexed format. This approach ensures you can access the data without restrictions from external services. Additionally, I wanted to avoid creating extra load on the free Overpass API.

Hosting OSM data on my server involves costs that I need to cover, so I charge a small fee for downloading more than 100 features. If the website gains more users, I may be able to reduce this price, as my goal isn’t to profit significantly from this service. 😊🤑

What’s Next for GeoDownloader.com

While I’m thrilled with how far GeoDownloader has come, I see this as just the beginning. Here’s what I’m planning next:

• Adding support for more file formats and data sources.

• Enhancing the filtering options with more granular controls.

• Increasing data limits for each package.

If you have ideas or features you’d like to see, I’d love to hear from you!

What I’ve learnt from this

Throughout the development of GeoDownloader.com, I gained invaluable experience and knowledge in handling OpenStreetMap (OSM) raw data. Here are some key takeaways from this journey:

1. Dealing with OSM Raw Data: I learned how to efficiently manage and process raw OSM data. This involved understanding the structure of OSM data and file format (PBF).

2. Converting Data with GDAL and Python: I utilized GDAL (Geospatial Data Abstraction Library) and Python to convert OSM data into an indexable format. This process included writing scripts to automate data conversion and ensuring the data was ready for further analysis and use.

3. Using GeoPandas for Data Conversion: I explored GeoPandas, a powerful Python library, to convert the processed data into various formats. GeoPandas made it easier to handle geospatial data and perform complex operations, such as reprojecting and merging datasets.

4. Building UI with AI Tools: I learned how to use AI tools, specifically Claude AI, to build user interfaces faster and more professionally. This significantly improved the efficiency and quality of the UI development process.

5. Integrating Stripe for Payments: As the server cost is too high to cover myself, I learned how to integrate Stripe to collect a small fee to cover these costs for large data sets. This integration was crucial for maintaining the sustainability of the project.

6. Using Tailwind CSS for Frontend Development: Although I am primarily a back-end developer and do not do much front-end coding in my daily job, I used the Tailwind CSS framework to build the UI. I found its ready-to-use class names extremely useful and efficient for developing interfaces.

Try It Out

GeoDownloader.com is live now, and I’d be honored if you gave it a try. I hope it makes your work with OpenStreetMap data faster, easier, and more enjoyable.

Head over to GeoDownloader.com and let me know what you think.

Adding a GeoJSON file as a layer is easy as drag & drop action

However, if the GeoJSON content you want to import is not in a file already, then you need to first create a file with this content and import it as shown above. This workaround could be time-consuming especially if you just want to visualize the GeoJSON to see where geometries are located on the map together with other layers repetitively.

On the other hand, you may ask why not just use geojson.io for this purpose. It is a valid question if your data is in WGS-84 (EPSG:4326) or Web Mercator (EPSG:3857) coordinate systems and you don’t want to see your data along with other layers, but if you need to visualize GeoJSON content in different coordinate systems with other layers, you had to follow the solution above until now.

During the holiday season, I developed a simple but effective plugin for this problem. Basically, you just need to paste your GeoJSON content into the input area, and that’s it. It doesn’t matter which coordinate system is in or contains different types of geometries. It recognizes different geometry types and creates layers for each type together with properties.

A screenshot from my Quick GeoJSON plugin

I don’t want to take your time, you can either search “Quick GeoJSON” in the plugin browser in QGIS or click here to see and install via the QGIS website.

I recently bought a Xiaomi Poco F3 (128GB) at £234 from Xiaomi UK website. I think it is great hardware at this price. I have been using iPhone for the last three years (XR->11->12). Once I saw this deal on hotukdeals website, I thought I should give it a try to Android after three years. I knew it will come with a lot of bloatware and I cannot uninstall them in a normal way, but I was ready to deal with it a hard way.

Before making the final decision for the purchase, I did a quick research on Google and I found some good articles that explain how to remove bloatware on the phone via ADB shell. It seemed straight forward and I decided to purchase the phone.

As soon as I got my phone, I followed these articles and started removing them one by one. However, I made a big mistake and suddenly my phone locked itself. I didn’t know that I should remove my phone from my Xiaomi Account before removing the Mi Account application. I had to call Mi Customer Support but they didn’t help at all, the guy on the phone asked me to reset my Mi Account password (didn’t understand why), once he realized that didn’t work, he asked me to send my identification documents to Xiaomi so that they could remove the lock. It was funny because I was removing these applications to stop sharing my personal data. After I did a bit of research on the internet, I found the solution. However, it was definitely annoying.

If you want to debloat your phone, I suggest this article that explains step by step.

Here is the list of apps that I deleted from my phone.

• com.android.chrome

  • Alternative: Microsoft Edge

• com.android.providers.downloads.ui

• com.android.providers.partnerbookmarks

• com.android.soundrecorder

• com.android.stk

• com.bsp.catchlog

• com.facebook.appmanager

• com.facebook.services

• com.facebook.system

• com.google.android.apps.googleassistant

• com.google.android.apps.messaging

  • Alternative: QKSMS

• com.google.android.apps.subscriptions.red

• com.google.android.apps.wellbeing

• com.google.android.calendar

  • Keep it if you use Google Calendar

  • Alternative: Microsoft Outlook

• com.google.android.contacts

  • Keep it if you use Google contacts
  • Alternative: Simple Contacts

• com.google.android.feedback

• com.google.android.gm

  • You may want to keep it if you use GMail
  • Alternative: Microsoft Outlook
  • Alternative: K-9 Mail

• com.google.android.googlequicksearchbox

• com.google.android.inputmethod.latin

  • It is Google Keyboard, you should install an alternative first otherwise it can break your system.
  • Alternative: Microsoft SwiftKey Keyboard

• com.google.android.marvin.talkback

• com.google.android.syncadapters.contacts

  • Keep it if you use Google Contacts

• com.mi.android.globalFileexplorer

  • Alternative: Cx File Explorer

• com.mi.android.globalminusscreen

• com.milink.service

• com.mipay.wallet.in

• com.miui.analytics

• com.miui.backup

• com.miui.bugreport

• com.miui.calculator

• com.miui.cleanmaster

• com.miui.cloudbackup

• com.miui.cloudservice

• com.miui.cloudservice.sysbase

• com.miui.daemon

• com.miui.freeform

• com.miui.gallery

  • Alternative: Simple Gallery Pro

• com.miui.hybrid

• com.miui.hybrid.accessory

• com.miui.micloudsync

• com.miui.miservice

• com.miui.mishare.connectivity

• com.miui.msa.global

• com.miui.notes

• com.miui.phrase

• com.miui.player

• com.miui.screenrecorder

• com.miui.screenshot

• com.miui.touchassistant

• com.miui.videoplayer

• com.miui.weather2

• com.miui.wmsvc

• com.miui.yellowpage

• com.tencent.soter.soterserver

• com.xiaomi.account

• com.xiaomi.glgm

• com.xiaomi.joyose

• com.xiaomi.mi_connect_service

• com.xiaomi.micloud.sdk

• com.xiaomi.midrop

• com.xiaomi.mipicks

• com.xiaomi.miplay_client

• com.xiaomi.payment

• com.xiaomi.scanner

• com.xiaomi.simactivate.service

• com.xiaomi.xmsf

• com.xiaomi.xmsfkeeper

As you can see, I put alternatives for some deleted apps because they are essential to read e-mails, browse the internet or your photo gallery. I listed Micorosft alternative for Google applications, you can ignore them if you are dependent on Google services. I mostly use Microsoft applications in my daily routine, so I didn’t want to have duplication of the same application from Google.

Unfortunately, this list doesn’t cover all installed bloatware because some applications are breaking the phone if you remove them. For example, I couldn’t manage to uninstall Security Center, Find Device service, and POCO Launcher. I really wanted to replace POCO Launcher with my favourite Android launcher, but it has not been possible so far. When you remove POCO Launcher, it breaks the Recent Apps button’s functionality and it stops working. Uninstalling the Security Center application (com.miui.securitycenter) or the Find Device service (com.xiaomi.finddevice) puts the phone in a boot loop.

In addition, when you get a new system update, you may find some uninstalled applications come back, thus I would suggest to keep the list of uninstalled applications to remove them again easily.

I noted some commands below that you may also find useful.

# List all installed packages
pm list packages

# List all installed and uninstalled apps
pm list packages -u

# Disable app
pm disable-user app.package.name

# Re-enable it
pm enable app.package.name

# Uninstall app
pm uninstall --user 0 app.package.name

# Install uninstalled app
pm install-existing app.package.name

#Enable ADB via Network
adb tcpip 5555

#Disable ADB Network
adb usb

# Diff Files
# It is usefull to compare output between (pm list packages) and (pm list packages -u), so you can keep the uninstalled apps list easily.
diff --changed-group-format='%>' --unchanged-group-format='' new_packages.txt  all_packages.txt

Python will be used to read SMS messages and forward them to Telegram Bot API. Messages will be listened to by Gammu SMS service and it will trigger the Python script when an SMS message received.

Setup

First of all, we have to enable communication between GSM hat and Raspberry PI Zero, run Raspberry Pi Software Configuration Tool (raspi-config);

$ sudo raspi-config

Then, follow following actions;

Go to Interface Options

Select and enter P6 Serial Port

Select No at this screen.

Lastly, select Yes at this screen.

Exit configuration tool. Now we need to enable UART on raspberry pi. Shutdown and eject SD card to open in another computer. Open /boot/config.txt file, find the below statement and uncomment it to enable the UART. You can directly append it at the end of the file as well.

enable_uart=1

Now, reboot raspberry pi device and open the console to install Gammu SMS Deamon and Python PiP package installer.

$ sudo apt-get update
$ sudo apt-get install python-pip gammu-smsd

Install Telegram Bot API python library.

$ sudo pip install python-telegram-bot==12.3.0

Note: Version 12.3.0 is the latest version that is compatible with Python 2.7, if you want to use it with Python 3+ version, you can install the latest available version.

Now you need to create a bot in Telegram and obtain a token. Afterwards, start a chat with that bot in your Telegram account and send a test message to your bot. This will help us to identify your chat ID.

Open following URL with you Telegram Bot token and copy the chat ID in the response;

https://api.telegram.org/bot[YOUR-BOT-TOKEN]/getUpdates

This is your chat ID.

Save following script in /home/pi/forward-telegram.py;

#!/usr/bin/env python
from __future__ import print_function
import os
import sys
import telegram

numparts = int(os.environ['DECODED_PARTS'])

text = ''
# Are there any decoded parts?
if numparts == 0:
    text = os.environ['SMS_1_TEXT']
# Get all text parts
else:
    for i in range(1, numparts + 1):
        varname = 'DECODED_%d_TEXT' % i
        if varname in os.environ:
            text = text + os.environ[varname]

# Log
print('Number %s have sent text: %s' % (os.environ['SMS_1_NUMBER'], text))

#Send by Telegram
bot = telegram.Bot(token='[YOUR-BOT-TOKEN]')
bot.send_message(chat_id=[YOUR CHAT ID], text=os.environ['SMS_1_NUMBER'].strip() + " | "+ text)

Make python file executable

$ chmod +x /home/pi/forward-telegram.py

Update Gammu SMS Deamon’s configuration to use GSM hat. Waveshare uses /dev/ttyS0 for communication.

$ sudo nano /etc/gammu-smsdrc

Find following lines in the configuration file and update with correct values, for me these values are following;

port = /dev/ttyS0
connection = at115200

Also, we need to specify the script path to be ran by Gammu so add following line below [smsd] section;

RunOnReceive = /home/pi/forward-telegram.py

You can find my complete configuration file here.

Restart Gammu SMS service.

$ sudo systemctl restart gammu-smsd.service

That’s all! You should now able to get your SMS messages to your Telegram account.

You can monitor Gammu SMS service for any problem with the following command;

$ sudo systemctl status gammu-smsd.service

az storage blob delete-batch -s mycontainer --pattern '*[/]*'

Assume that your report uses DirectQuery and API document says you cannot use Update Parameters endpoint. In this article, I presume that the report uses Direct Query with MS SQL Server. The only thing I can change via APIs is the connection parameters via Update Datasource endpoint. Luckily, SQL Server 2016 introduced with Row-Level Security that allows us to limit user’s access to data at row level. We can create a master table to hold master data for each project, and the rest of the tables can make inner join with this table for Power BI reports. During the CI/CD process, we can call Update Datasource endpoint for the newly created dataset and update its user credential to limit its data for a specific project. It can be used as a solution until Microsft removes restrictions on Update Parameter’s endpoint for Direct Query based reports. This solution makes a bit complicated, but it works. Hope you find it useful. Leave your comments if you have better a solution, love to hear it.

First, I created an API key via Bunq’s mobile application. Then, I started to read API documentation to find out which endpoints are available to export statements.

I found that there is an endpoint with /user/{userID}/monetary-account/{monetary-accountID}/customer-statement, it generates statements and returns statement’s Id, so that I can download the statement with this ID. It also requires userID and monetary-accountID as route parameters.

Bunq provides Python SDK; this makes my job easier for authentication and API calls.

I started to code, first of all, I need to create an API context with following;

from bunq.sdk import context, client

apiContext =  context.ApiContext(context.ApiEnvironmentType.PRODUCTION, '[API KEY]',  "ExportStatementClient")
apiContext.save('D:/BunqApiContext')

Then, I need to obtain my userID, and I can get it with the following code;

rom bunq.sdk import context, client

apiContext =  context.ApiContext(context.ApiEnvironmentType.PRODUCTION, '[API KEY]',  "ExportStatementClient")
apiContext.restore('D:/BunqApiContext')
bunq_client = client.ApiClient(apiContext)

response = bunq_client.get('user', [], [])
print (response.body_bytes)

I noted my userID in response to use it later.

Now I can get my monetary-accountID with following;

bunq_client = client.ApiClient(apiContext)
response = bunq_client.get('user/{userId}/monetary-account-bank', [], [])
print (response.body_bytes)

It returns all your accounts, so note your monetary-accountID in response to export its statements.

Now, as we have all the required parameters, we can export statements as PDF files for the last 90 days in one go without using dealing with multiple UI actions for each of them.

import requests
from datetime import date, timedelta
import time
import uuid
from bunq.sdk import context, client
import json

def dict_to_bytes(the_dict):
    return bytes(json.dumps(the_dict).encode('utf-8'))

if __name__ == "__main__":
    apiContext =  context.ApiContext(context.ApiEnvironmentType.PRODUCTION, '[API KEY]',  "ExportStatementClient")
    apiContext.restore('D:/BunqApiContext')
    bunq_client = client.ApiClient(apiContext)

    userId = XXXXX
    accountId = XXXX
    output_folder = 'D:/BunqStatements'

    today = date.today()
    first_date = today + timedelta(days=-90)
    current_date = first_date
    i = 0
    statements = []
    while i < 90:
        params = {
            "statement_format": "PDF",
            "date_start": current_date.strftime("%d-%m-%Y"),
            "date_end": (current_date+ timedelta(days=1)).strftime("%d-%m-%Y")
        }
        response = bunq_client.post('user/'+str(userId)+'/monetary-account/'+str(accountId)+'/customer-statement', dict_to_bytes(params), [])
        statement_id = json.loads(response.body_bytes)['Response'][0]['Id']['id']
        statements.append(statement_id)
        response = bunq_client.get('user/'+str(userId)+'/monetary-account/'+str(accountId)+'/customer-statement/'+str(statement_id)+'/content', [], [])
        f = open(output_folder + '\statement-'+current_date.strftime("%Y-%m-%d")+'.pdf', 'wb+')
        f.write(response.body_bytes)
        f.close()
        bunq_client.delete('user/'+str(userId)+'/monetary-account/'+str(accountId)+'/customer-statement/'+str(statement_id), [])
        i = i + 1
        current_date = first_date + timedelta(days=i)
        print (current_date.strftime("%d-%m-%Y")+' saved.')

That’s all!

I attached them to my visa application as supporting documents, and eventually, my visa application was approved.

import sys
from pyspark.sql.session import SparkSession

def get_spark(jdbc_driver_path):
    return SparkSession.builder.master("local[10]").config("spark.driver.extraClassPath", jdbc_driver_path).getOrCreate()

def get_sql_dataframe(host, database_name, table_name, order_by, upperBound, partition_num):
    return spark.read.format("jdbc")\
        .option("url", "jdbc:sqlserver://{host};databasename={database_name};IntegratedSecurity=true".format(host=host, database_name=database_name))\
        .option("dbtable", "(SELECT ROW_NUMBER() OVER (ORDER BY {order_by}) AS row_num, * FROM {table_name}) as tmp".format(order_by=order_by, table_name=table_name))\
        .option("partitionColumn", "row_num")\
        .option("lowerBound", 0)\
        .option("upperBound", upperBound)\
        .option("numPartitions", partition_num)\
        .load()\
        .drop('row_num')

def get_count(host, database_name, table_name):
    return spark.read.format("jdbc")\
        .option("url", "jdbc:sqlserver://{host};databasename={database_name};IntegratedSecurity=true".format(host=host, database_name=database_name))\
        .option("dbtable", "(SELECT COUNT(*) as row_count FROM {table_name}) as tmp".format(table_name=table_name))\
        .load()\
        .first()['row_count']

def export(sql_df, output_folder):
    sql_df.write.mode("overwrite").parquet(output_folder)

if __name__ == "__main__":
    # Database Configuration
    host = '[Host Name]'
    database_name = '[Database Name]'
    table_name = '[Table Name]'
    order_by = '[Indexed Column] DESC'
    jdbc_driver_path = 'mssql-jdbc-6.4.0.jre8.jar'

    # Output
    output_folder = '[Output Folder]'
    partition_num = 1000

    spark = get_spark(jdbc_driver_path)

    total_count = get_count(host, database_name, table_name)

    print str(total_count) + " rows will be exported."

    sql_df = get_sql_dataframe(host, database_name, table_name, order_by, total_count, partition_num)

    export(sql_df, output_folder)

from pyspark.sql.functions import col, dayofmonth, month, year

df = #load data frame

dt = col("date").cast("date")
fname = [(year, "year"), (month, "month"), (dayofmonth, "day")]
exprs = [col("*")] + [f(dt).alias(name) for f, name in fname]

(df.select(*exprs).write.partitionBy(*(name for _, name in fname)).mode("overwrite").parquet('output_location'))