Show HN: OpenFreeMap – Open-Source Map Hosting
openfreemap.orgHi HN,
After 9 years of running my own OpenStreetMap tile server infra for MapHub (https://news.ycombinator.com/item?id=11389989), I've open-sourced it and launched OpenFreeMap.
You can either self-host or use our public instance. Everything is open-source, including the full production setup — there’s no 'open-core' model here. Check out the repo (https://github.com/hyperknot/openfreemap). The map data comes from OpenStreetMap.
I also provide weekly full planet downloads both in Btrfs and MBTiles formats.
I aim to cover the running costs of the public instance through donations.
Looking forwards for your feedback.
Something I'd be willing to pay for is historical tiles/data.
I'm writing an application that lets you view historical data, and the problem is that most/all map services show only current data. It would be nice if the map reflected the year of the data we're layering on top.
So if you want to make some money, there's an option for you!
Have you heard of:
https://www.openhistoricalmap.org
I don't think you can pay for it, but I'm sure they'd accept donations!
Interesting but not entirely accurate. Moldova has not existed as a state until 1991, after the dissolution of the USSR but it shows up on the map after WW2.
The great thing about open map projects is that they can be updated, and store a historical edit log along with discussions and sources for each change.
As with other community projects such as Wikipedia or OSM, OpenHistoricalMap relies on keen-eyed volunteers such as yourself to improve their database!
Ah, wonderful -- thank you
The key problem here is a lack of accurate data. OSM is amazing, but is constantly getting improved and closer to realtime "ground truth." Historical OSM data is usually worse all-around, rather than "equally good, but as of year XXXX"
Perhaps there is a different dataset with better historical accuracy than OSM, but I do not know of one. You'd likely have to reduce your scope: trails in a specific park? Roads in a specific city?
As another commenter notes, there is a sister project OpenHistoricalMap that focuses on mapping historical data. It is rather sparse compared to OSM, but super fun to explore around e.g. NYC.
As a hobby historian (not really haha) I'm interested in old streets and names particularly.
Especially here in Europe / Germany, where entire quarters where bombed to rubbles newer streets do not match neither namely nor spatially.
I loved how you could "move back in time" in street view. I think that has been killed too? There is a lovely Twitter/X account for Detroit tho: https://x.com/DetroitStreetVu
You can still go back in time on Street View. There's a "See more dates" link next to the address info in the top left (at least on the desktop Maps interface) that when you click it opens a film strip of different dated captures for that location. Here's the 2007 capture of 1 Embarcadero (outside the Ferry Building in San Francisco) for example: https://maps.app.goo.gl/ArrucFgus9uMvdSaA
Oh yeah, true. Thanks for pointing out.
It's just that in Europe the data is much more sparse to begin with. In many places Google didn't go multiple times, so there just can't be older data. On some streets however I see that if the data was really old (before ~2010/2012) Google probably decided that the quality was so bad – well, it really is – that they do not make it available. So even if there were multiple passes it is not a must that this feature comes available.
Maybe not exactly what you had in mind (sounds more like a street view-like timeline of changes, if I understand you correctly), but
https://maps.arcanum.com/
has a great selection of historical maps.
I think you’d be interested in Pastmaps:
https://pastmaps.com
I really like the idea.
Why OMT instead of protomaps? The latter is clearly where the community is moving towards (albeit very slowly).
I'm somewhat sceptical about the "free with no API keys" idea. I guess your service is not guaranteed to be up so no one too big will rely on it. But what if you start getting abuse or someone using them on some humongous site (e.g. one of those cheap restaurant email builders that always embed a map), and you start getting way too much traffic from random sources and websites. What would you do?
From the FAQs on GitHub [1]
> What about PMTiles?
> I would have loved to use PMTiles; they are a brilliant idea!
> Unfortunately, making range requests in 80 GB files just doesn't work in production. It is fine for files smaller than 500 MB, but it has terrible latency and caching issues for full planet datasets.
> If PMTiles implements splitting to <10 MB files, it can be a valid alternative to running servers.
[1] https://github.com/hyperknot/openfreemap
See my response here: https://news.ycombinator.com/item?id=41638031
That's an interesting claim. I make range requests to 100GB+ files (genomics) all the time for work and it works great. I've never considered total file size as directly related to latency in this respect, assuming you have some sort of an index of course.
You can test this claim directly against a AWS S3 bucket.
First 100KB of a 100GB+ file:
curl -H "Range: bytes=0-100000" https://overturemaps-tiles-us-west-2-beta.s3.amazonaws.com/2... --output tmp -w "%{time_total}"
First 100KB at the 100GB mark:
curl -H "Range: bytes=100000000000-100000100000" https://overturemaps-tiles-us-west-2-beta.s3.amazonaws.com/2... --output tmp -w "%{time_total}"
Here the requests are really really small, on average 405 bytes each. I guess in your genomics work you are making larger requests, so probably it's not so much of an issue.
BTW, we are discussing latency with bdon in this issue, it seems to be specific to Cloudflare: https://github.com/hyperknot/openfreemap/issues/16
I just tried @bmon's curl examples above with 100 byte requests. Similar results. I think the Cloudflare explanation is more likely.
If you store the PMTiles in S3 or any other object store that supports HTTP Range Requests, that's a no-brainer... In a normal disk on you own server, this might become interesting, yes.
ok except "full planet datasets" make little sense for terrestrial features. Splitting .. aka sharding the files into basic continents would make SO much sense. Asia is big, but no requests for Africa mixed in.. Australia would be manageable?
PMTiles could come up with a version in the future where instead of one 90 GB file, they have 9 thousand 10 MB files. That would work well I believe.
The latency for small files and ranges of large files is pretty similar on most storage platforms, but there are some exceptions like Cloudflare R2.
The main reason PMTiles is one file and not two or more files is that it enables atomic updates in-place (which every mature storage platform supports) as well as ETag content change detection in downstream caches. All of the server and serverless implementations at http://github.com/protomaps support this now for AWS, S3-compatible storage, Google Cloud, and Azure.
Now I'm curious, what causes the latency for range requests with R2?
I don't have any insight into this other than observing how their storage system works, but here's some scripts I made last year to test:
https://github.com/bdon/cloudflare-r2-latency
Range requests means work and logic. Getting a file requires no logic.
Also, I'm pretty sure range requests are going to be difficult to cache. That implies going to origin every request which is bad.
So Protomaps is a really full stack map platform, they are competing with kind of like 3-4 projects at once:
- There is the Protomaps schema. It's competing with OpenMapTiles and Shortbread (https://shortbread-tiles.org/)
- There is the Protomaps file format. It's competing with MBTiles and say Btrfs images in OpenFreeMap.
- There is the Protomaps styles. It's competing with OSM Liberty and OSM Bright styles (https://github.com/maputnik/osm-liberty)
- There is the way to host the tiles, on CloudFlare workers, range requests, etc. It's competing with MBTiles servers or OpenFreeMap's nginx servers.
About getting abuse, how do you imagine it gets abuse? It's a static file showing OpenStreetMap data.
I mean traffic is the only thing what can be a problem, but I think Hetzner should be good till 30-100 TB per server.
1) The Protomaps schema is mostly a re-implementation of the Tilezen project https://tilezen.readthedocs.io/en/latest/ which is a linux foundation project. OpenMapTiles, which OpenFreeMap uses, while open source, does not have a license that encourages derivative works or enables distributing styles under a standard FOSS software license (https://www.npmjs.com/package/protomaps-themes-base). That's also one motivation for developing Shortbread which is at this point less developed than Tilezen.
2) The file format (PMTiles) addresses a different audience than either MBTiles or Btrfs images. Both of those require administering a server for tiles, while PMTiles requires static blob storage and nothing else. You do have the option of using a server like MBTiles/btrfs which ought to be comparable in latency, and that's documented here: https://docs.protomaps.com/deploy/ as well as Lambda, Cloudflare Workers, Google Cloud Run and Azure Serverless functions.
3) There are no existing styles for MapLibre GL that work off the Tilezen layer, generalization and tagging scheme, so we need to develop one style, with multiple themes.
> OpenMapTiles, which OpenFreeMap uses, while open source, does not have a license that encourages derivative works
Can you elaborate on this ? I'm derivating OMT and am quite worried now ^^
OMT use a CC-BY license: https://creativecommons.org/faq/#can-i-apply-a-creative-comm... (edit: link)
This means that software that implements OMT, even if written from scratch, cannot be re-used by other FOSS projects (Apache, BSD, GPL, AGPL, other software in the OpenStreetMap ecosystem, etc) without affecting the license.
Ideally for Protomaps it should be possible to re-use just one portion - like only the label layer with your own layers from other sources, or even bundle it as a JS dependency in another open source project - without affecting the license of downstream projects.
Are you talking about just the OpenMapTiles spec, or some adjacent software? I'm certain that you can build software to some specification without ever agreeing to the spec text's licence, and that a CC-BY licenced spec doesn't limit any implementing system's licence.
Even so, CC-BY is permissive and you could include CC-BY content in a, say, GPL project. You just need to include both licences.
> I'm certain that you can build software to some specification without ever agreeing to the spec text's licence
That is exactly the opposite of OMT's copyright interpretation: https://github.com/openmaptiles/openmaptiles?tab=readme-ov-f...
> You just need to include both licences.
That is the definition of license incompatibility as described in the Creative Commons documentation above. The license is open source and a good fit for if you are running a paid map SaaS or free service as an end product, but is not compatible with the open source ecosystem as a building block.
> That is exactly the opposite of OMT's copyright interpretation.
They are entitled to their opinion, but I don't consider a database to be a derivative work of a schema (this should be clear, since databases of facts aren't copyrightable anyway), same for software (it's at least fair use in the US, see Google v. Oracle). This goes back to the debate on copyrightability of APIs, where a decision for copyrightability would ruin swathes of the software industry and much of free software.
Though I understand if you'd rather avoid using projects by people with weird legal opinions.
> That is the definition of license incompatibility as described in the Creative Commons documentation above.
Where do you see that? All I found was a statement that the CC share-alike licences can be converted to GPL, but no word on non-SA licences:
> Version 4.0 of CC’s Attribution-ShareAlike (BY-SA) license is one-way compatible with the GNU General Public License version 3.0 (GPLv3).
The FSF says that CC-BY is compatible with the GPL, and I believe this extends to all reasonable FLOSS licences: https://www.gnu.org/licenses/license-list.html#ccby
It's not uncommon for a piece of code to have multiple licences in a big project. One for licencing to that project, and another for sublicencing to the end user. Proprietary apps that include FLOSS code do this all the time.
> Where do you see that?
"Additionally, our licenses are currently not compatible with the major software licenses, so it would be difficult to integrate CC-licensed work with other free software. Existing software licenses were designed specifically for use with software and offer a similar set of rights to the Creative Commons licenses." https://creativecommons.org/faq/
> Though I understand if you'd rather avoid using projects by people with weird legal opinions.
The scale of the Protomaps project is small (I'm the only full-time developer) and I don't have the resources to interpret novel copyright situations. I think it's best for the ecosystem to abide by the license terms stated by the open source developer, instead of challenging their validity.
I did 150TB+ on a 40 Euro Hetzner server for years without any issue and also did the same amount in a similar price range on multiple different providers. The cloud and CDNs charge a huge markup.
That's great to know! I know of one story where someone got terminated on Hetzner because of the bandwidth but it turned out he was also running a Tor node.
Related : I'm building https://github.com/laem/cartes, an alternative to Google Maps.
On top of a custom protomaps tileset and a few other MapTiler options (such as satellite and hiking), it packs a search engine, a basic OSM place UI, transit calculators (walk, bike with profiles, transit, car), small features like ruler and favorites, transit maps, photos of places, place search by categories, and the French open source street view Panoramax.
Of course, given the scope, its alpha software.
It's built locally for France and French speaking users, though most of the code is English, some data sets are not. I'm spending ~ 50 % working on transit, lots need to be done.
You can test it here : https://cartes.app.
The aim is not to provide map tiles as an API to other project, but to build a UI on it.
Wow this is one of the best Google Maps clones I've seen so far. Is there a plan to translate the UI into other languages than French? I would be willing to help out, if there is a Weblate interface or something similar.
Thanks ! I've raised this issue. Short : nothing yet.
https://github.com/laem/cartes/issues/574
I see you're sourcing images from Wikimedia. Have you found any other good sources? I'm working on indexing images by place taken for my own project and am primarily very slowly and jankily enumerating the flickr API. (You can see https://plantopo.com/geophotos for a pre-alpha view of where I have coverage. Zoom in where there are blue dots and click the map to search. Don't click Oregon or Washington because of a bug I'm working on)
Are you using opentripplanner for routing?
Wikimedia is a totally underrated source of photos.
The problem is filtering good ones. See my PR of the day, where I'm using only the images that are featured on Wikipedia pages. It gives better results than google maps in lots of places. Also some weird pictures like logos but I'm confident they can be ruled out.
Your map is cool !
https://plantopo.com/geophotos#13.92/50.91621/-1.37752
https://cartes.app/?photos=oui#13.92/50.91621/-1.37752
With this second link you'll understand the why of my PR : there's just too many Wikimedia photos ^^
Edit : PR merged, you'll see less photos in 1 minute https://github.com/laem/cartes/pull/611
Interesting! I'll definitely check out Wikimedia.
My project is for planning hiking trips so the areas I'm searching typically are places in the wilderness that don't have many photos. Right now I'm not doing any ranking. I just semi-randomly take ten photos for the search area.
> Are you using opentripplanner for routing?
No, Motis. I picked the first I could install on my computer at the time. Turns out, it's a very promising project.
But switching to OTP wouldn't be such a problem. GTFS in, JSON out. It's a one day work change for basic functionnality.
Oh, there is another pal who's streaming making his own maps, fairly entertaining: https://youtube.com/playlist?list=PL980gcR1LE3L8RoIMSNBFfw4d...
Thanks ! I've thought about making an "education" project but it takes time, and after 9 months of working on it I'm now convinced it's possible to build a basic Gmaps alternative in a few years. Business model still missing though !
Edit : just watched some minutes, he's rebuilding low-level components, like route finding. Impressive stuff but not quite user-friendly.
> not quite user-friendly
Yes, it does not seem like he makes a business or a product for a wide audience. It's more about exploring the unknown, teaching programming and being entertaining.
Although I adore such ambitious projects that seem insurmountable at the first glance.
Wow, now that's an ambitious project! I wish you a lot of success with it!
Thanks ! It consists of using more focus initiatives like yours and bdon's :)
Neat!
We used to host our own mbtiles map for like $11/mo, but the problem was (this was prior to planetiler and people generating public worldwide mbtile dumps) there wasn't a free/cheap source for regularly-updated mbtiles. The dump from OpenMapTiles was not updated for years.
So we gave up and went to mapbox, where we regularly exceed the monthly free tier for web, but they give us a discount. Because that is a scary scenario and we are dying to pay a fixed monthly fee, I think we will try yours and donate!
If your application allows for it, I also recommend checking out Protomaps / PMTiles. Brandon has some insanely cool work making a tile format and related infrastructure that scales nicely.
I wanted to, but I was unable to get Protomaps working in our application, unfortunately. I'm not much of a js person (or a dev) and it was a bigger change than I could apparently handle in our rails app. Actually, thinking back on it, I think the main issue - or at least the one immediately presenting itself - was a CORS problem (a friend was hosting the PMTiles in their DO Spaces). And then I ran out of patience.
Thanks for giving it a try! If you're going to revisit this, we could improve the docs around setting CORS configuration on DigitalOcean: https://docs.protomaps.com/pmtiles/cloud-storage#digitalocea...
Yeah there are some pretty specific requirements for the server hosting the PMtiles that weren't well documented when I last looked at it.
Look into Planetiler [1]. It's a resource-efficient and easy to setup project to generate mbtiles or pmtiles from OSM pbf files, which are free to download from Geofabrik. It uses the openmaptiles spec for the result files and you can also generate some custom tiles by writing yaml (or forking the java source).
Protomaps also provides daily updated full-planet pmtiles [2], from which you can create a pmtiles extract using its CLI.
[1] https://github.com/onthegomap/planetiler [2] https://docs.protomaps.com/basemaps/downloads
To be clear, OpenMapTiles is kept up-to-date. The latest version was published in April: https://www.maptiler.com/news/2024/04/openstreetmap-data-pre...
I think he was referring to a freely available full planet OSM mbtiles dump, to be used with OpenMapTiles. I cannot find a reference to that in your blog post.
Ha, yes, obviously I was referring to the free mbtiles download. Maptiler is currently showing "2020 OpenStreetMap Vectors only"
Thank you for sharing the story! I hope OpenFreeMap can contribute either with the MBTiles / Btrfs dumps, or with the public instance.
Thanks for offering the service! The intensity of map tile pricing is always a little baffling to me. For example, with mapbox this month we (got linked to on The Verge and) exceeded the free tier on the 22nd. Two days later and we have a $40 bill and counting, with a week left in the month. We're a free, open source project and can't handle dynamic pricing like this.
My only feedback on your tiles is the lack of POIs. Mostly just bus stops and an occasional supermarket in my area. But I see from the github that this is on the roadmap.
Thanks a lot! Yes, I believe this dynamic pricing is making it impossible for small or non-profit projects to host maps. And you can see, "fun" maps basically disappeared from the internet - they were really common 10 years ago!
For POIs, they are present in the data, it's just that the styles might not have the matching icons for them. But you can always make them into a circle, etc.
Having said that, working on the styles is a big and never ending effort, so if someone would like to help with matching the POIs then contributors welcome.
This is really cool, I'll definitely look at this if I need to embed a map in the future. Are there any libraries that lets you plot geographic data on top of this in Python, with Matplotlib or similar?
Slightly off topic, but how come there doesn't seem to be any open projects using the Overture maps data?
> Overture is a data-centric map project, not a community of individual map editors. Therefore, Overture is intended to be complementary to OSM. We combine OSM with other sources to produce new open map data sets. Overture data will be available for use by the OpenStreetMap community under compatible open data licenses. Overture members are encouraged to contribute to OSM directly.
Sounds like it would be a good data source, or am I missing something?
https://overturemaps.org/about/faq/
Overture is an awesome resource because it has more coverage than OSM, but at the cost of accuracy. When I was considering using it for restaurants in surprisedatespot.com, I found that it has a lot more automated content with bad geocoding (imprecise locations) compared to OSM.
E.g. restaurants can come from an old Facebook page and the geocoding might not be accurate compared to a survey.
So where OSM might have a restaurant that closed 8 years ago, overture has every restaurant that's been at that location for the past 8 years.
Folium is a useful library for plotting geographic data in Python. It generates a webpage with the map data displayed with Leaflet.js
https://python-visualization.github.io/folium/latest/ https://leafletjs.com/
I develop Lonboard [0], a Python library for plotting large geospatial data. If you have small data (~max 30,000 coordinates), leaflet-based Python libraries like folium and ipyleaflet can be fine, but because Lonboard uses deck.gl for GPU-accelerated rendering, it's 30-50x faster than leaflet for large datasets [1].
[0]: https://developmentseed.org/lonboard/latest/
[1]: https://developmentseed.org/lonboard/latest/how-it-works/
I understand you don't plan to make money, only to cover costs. Nevertheless, I believe you would benefit from a more explicit "business plan". Your Gold support plan kind of fulfills that purpose (email support and an invoice), but right now it's kind of hidden in the middle of a donation request, and flavored as a donation. You might convince a lot more mba-type people to support you if you also extend the same offer under a heading like "business plan" (where you make it clear that commercial use is free, but you can subscribe to this totally-not-a-donation plan for email support)
Do you think it'd be more clear to only accept donations on GitHub Sponsors and only offer actual support plans on the website? I mean actual email support, helping companies migrate their existing map stack to OpenFreeMap.
If you aren't doing this for a living and getting paid for it, or having someone else do it, you should reconsider either A) providing the support or B) how you make money.
You may think it noble to "only take donations" just to pay the hosting cost, etc. but providing people support is a completely different thing. People are stupid, needy and inconsiderate of your time, feelings and experience.
If you don't want to run ads, great! But your time should be compensated fairly as well. You should be charging businesses for things, with a substantial cost for substantial work. Charging a business is not the same as charging an end user and you should not be averse to doing so.
100% keep sponsorship separate from paid support.
Thanks, I've thought about it and rewritten the whole donations / sponsorships / support plan section. What do you think?
Would be super cool if you have a bitcoin and/or monero option. Even if just address for donations and not as payment for support.
The problem with putting different options in different places is that a lot of people will only ever see one of the options.
I second making a very generous free tier, and then charging reasonably for more requests in wide buckets.
As a user, free or paid, it'd give me a lot more faith in this living past tomorrow.
I definitely not want to charge for requests, that's the founding principle of this project.
And the risk of this project is largely mitigated by offering the full repo as open-source, so you can always self host this if needed.
If you don't want to charge per request, you could also offer two pools of backend servers: free servers that are provided on a best-effort basis with fair-use-policy, and paid servers that are exclusive to people in the $150/month plan.
The latter should be easier to manage with more predictable traffic, while giving you more leeway to deal with abuse on the free servers if it ever becomes an issue.
Yes, I was thinking about that, it might actually be a good idea. Provide a premium service with SLAs, while providing the SLA-less service for free, with two pool of servers.
To be honest I don't see the problem with making commercial use non-free.
(a) They're making millions, you deserve a cut of that.
(b) If making money makes you more likely to be around in 10 years, that's attractive to businesses. Nobody knows when your donations might suddenly dry up.
The thing is that hosting this really isn't that expensive. My first GitHub Sponsors goal is to reach $175 per month, which would cover the hosting on Hetzner. Of course this cost can grow if the project gets more popular, but it's a good start!
I honestly believe this project can cover $175 per month eventually.
Of course there is the countless hours which went and which can possibly go into this, but to cover the hosting is still meaningful.
I mean, sure, but
Let's say 1 year from now your day job gets toxic and you have to work 18 hours a day just to pay your rent and expenses. Are you still going to maintain this?
What if the economy goes to shit and you have trouble scraping that $175/month? How long are you going to pay $175/month for something that isn't making money?
What if, instead, this became a part-time job of yours?
If I was a commercial user of your product, some assurance that your random personal situations aren't going to affect the product stability for a long time would be a good thing.
Yeah it is way easier for someone at a company to get approval for a paid service than for a "donation" of any kind, which raises lots of red flags.
This is neat! After the huge price increase a couple years ago with GMaps, it looks like you’ve come up with a creative way to offer much of the same functionality for most users. It’s unfortunate to still see small business sites to this day with error messages on their map widgets because of the API change from Google.
I know nothing about mapping, but I’ve always had a dream of making a private neighborhood map which labels each house with people, sort of like a visual Rolodex. It would be great to have property boundaries, too. Do you have any suggestions for places to start? I don’t even know what file formats would hold this sort of data.
If you live in the US, you can get parcel information from your local tax assessor's website, usually. Sometimes it's devoid of owner information. Such cases usually require going to a company like ReGrid or Corelogic to purchase the data.
I work for a company (onXmaps) which produces user-friendly maps of such information. You could use our app to create such a "rolodex" of your neighborhood, by annotating individual parcels with your neighbors' information.
https://onxmaps.com
Thanks, Tony! I’m actually an Elite member of the off-road flavor! Is your recommendation to essentially use waypoints for the info?
In my mind I’d really like to host my own map which neighbors could share, just starting from a baseline of parcel boundaries and a way to drop info on each plot (onX does this wonderfully already).
Back in the day, my mom would make a little neighborhood diagram using MS Publisher and drawing boxes at the approximate locations of houses, and include names & phone numbers of each house. She would then distribute copies to the neighbors to hang on fridges. I loved that idea and basically want an online version of that, in a perfect world!
nice! yes, that's what I was suggesting for sure. You could put waypoints with contact info in a folder, and share that? Wouldn't obviously be useful for people without the apps.
Thanks! I don't know where you'd be able to get property boundaries, I guess it's usually not open-source / public data? Probably depends on the country / city.
In the USA, parcel outlines are commonly handled at the county level. You can often do a search for “[your county] GIS maps” and find their ArcGIS map of the county property lines with metadata about who owns what. Who knows if any of them are willing to share the data, but that’s where it lives.
this might be helpful?: https://github.com/martynafford/natural-earth-geojson/blob/m...
public domain map data in GeoJSON format.
also, for google maps I know you can respond to interactions with the supplied points of interest like buildings, including homes, and from that point you can obtain an address via reverse geocoding or some other technique. I expect other mapping tools may have similar abilities but I don't have any experience with anything other than Google.
FYI if you need a CDN you can get bulk pricing from fastly with a monthly commitment.
I'm not sure if their hosting works for you because you might be serving byte ranges. I think we're paying 2k/month usd for like 300TB/month. Backblaze-to-fastly is free, so no egress.
Love your project, more power to you. I'll be using this for a few side projects for sure.
At Hetzner, the bandwidth is unlimited in theory. In practice, there are stories of terminating accounts of those who used too much.
I don't think this project will ever make $2k per month. TBH, I'd be happy it it ever reaches $200 per month. I have to be very careful about what hosting providers to use and CDNs are out of the question.
I don't think it's still offered anywhere in the web interface, but Hetzner offers servers with 10Gbit uplink, with traffic charged at 1€/TB. Depending on resource usage this can be more cost effective than renting multiple servers, and paying for your traffic should prevent account termination for excessive traffic use.
https://docs.hetzner.com/robot/dedicated-server/network/10g-...
If I were running fastly, I'd donate bandwidth so that you would implement turn-key, bring-your-api-key fastly support to your open source setup!
As for other cdns - I remember https://neocities.org/ built their own cdn - but I couldn't immediately find a blog post about the cdn, or something related in their GitHub (unless every node runs part of it, which would make sense, I guess).
That's true: they do sponsor projects, as does cloudflare.
They could use you as a PoC/demo for their serverless stuff. Same with cloudflare.
It's worth it to ask. I can send you the contact for our fastly rep if you'd like.
The problem with hetzner, or any point solution, is that at some point you will saturate your bandwidth - either on the network or storage side.
Maybe it's worth front-ending it with cloudflare free, just to see/measure cache hits/requests/bandwidth?
Cloudflare Free is a good idea, but I'm afraid they'd be the one who limit me before. There is a comment in this thread who did 150 TB+ on Hetzner. I don't believe Cloudflare Free would allow 150 TB, not alone 300 TB in a month (I use two servers currently).
I wonder if other CDNs can help.
In addition to Cloudflare, bunny was on the radar too.
Bunny is also $10-$60/TB/month.
I believe Hetzner should not have any problem with using 100 mbps out of the 1 Gbps on their servers. That's 30 TB/month for free, per server. It'd be $300-$1800 per month on Bunny.
For this use case i would use the bunny volume tier which is $5/TB
That's still an order of magnitude more than what you can get with rented servers. I pay between 0.2 and 0.4 per TB doing nearly 2000TB a month.
Agreed. Getting a dedicated fibre link colocated is a great way to backfill and push the free tiers of CDNS.
Just like the cloud becoming easier to run on your own environments, tons of bandwidth is accessible too.
I think this is great, and also that you are getting some good advice in the comments about covering costs and/or monitization. I'd add my thoughts, but others have already expressed them here so I just wanted to say I love seeing this kind of philosophy expressed in a project like this.
edit to fix typo
If it does a good job, somebody should get paid for it. I wouldn't rely on using this if it depends on donations (I call this begware). People expect to pay for services and products.
My main project is MapHub, it gives me my income. It's not an income like you'd get in a FAANG job, but still, I'm running it for making a living.
For MapHub to work, I had to invest in my own tile server infrastructure: there is no way I could have made MapHub financially profitable with a 3rd party tile provider. So I kept iterating and iterating on the tile server infrastructure and after 9 years I arrived at a state which was worth open-sourcing.
I'm happy if this project gets to the point from sponsorship where my hosting bill is covered ($175 per month), but it doesn't have to be financially successful on its own.
I'll keep working on it because it's part of my main business, not because I beg for donations.
Have you been running this for long? I've looked for something in this space for the last couple of years but only came across MapTiler /AGOL and a few others, that were either pretty expensive (for small datasets) or restrictive functionality.
If so I'd say you need to work on the SEO and findability side.
I've literally just launched it.
As long as it stays with one server that’s fine. But what do you do in the hypothetical scenario that Facebook decides to use your tile server?
Nginx is fast, but I think that’s asking too much of it.
Of course that problem hasn’t happened yet, but it’s what I’d be worried about.
I benchmarked and nginx can host 30 Gbps on cold cache, on actual real world load. (I played back a log of 1 million real world requests).
So using nginx in a 1 Gbps server is definitely not asking too much of it, or I might not have understood your comment clearly.
Why would Facebook decide to use my tile server? When a company's business depends on having reliable map tiles, they either self host or have an Enterprise deal with Mapbox, etc.
I just used it as an example of some large service. The point is that I expect someone will take advantage.
It’s interesting to know that the limiting factor is bandwidth and not nginx/disk speed though. I’ll admit I don’t see 30Gbit of map tiles happening any time soon.
You are right in that I'd be afraid if this was hosted on AWS or on Cloudflare Workers. This whole idea is only possible because there are some hosts like Hetzner, which offer flat rate bandwidth.
I have actually worked on Facebook's mapping infrastructure. They moved away from Google Maps and Mapbox because they wanted maximum control.
The specific example you give is not going to happen.
> I call this begware
So disrespectful and so ignorant. You rely on free software without even knowing it but are happy to not dig around much and benefit from it. But heres a new project and you call it begware.
If you have nothing nice to say, don't say anything at all.
In that case just use one of the many map tile providers already out there. This is huge for people that don't want to pay but also can't justify using that much server resources just to self-host.
it says you can host it yourself and that he may offer a paid hosted plan in the future
> (I call this begware)
This only reflects how fucking stupid you are. If you read the post in any detail, you'd realize the author has been running a paid service for a while (9 years) and does not need your help.
I absolutely love this project and had a similar idea on my todo list for a while because I agree there should be at least one free open map provider with a developer friendly stack.
If you want free servers/bandwidth around the world, hit me up. My email is in my profile.
Thanks a lot for the offer, I'll definitely reach out!
Alternative: https://versatiles.org/
Just saw a german Talk about this yesterday.
Versatiles provides a free and open to use Server under tiles.versatiles.org which is sponsored by People around the CCC.
Versatiles is also the First to offer a free to use vector card server, which is also amazing.
(For the German Readers: https://media.ccc.de/v/ds24-406-versatiles-freie-karten-fr-a...)
Yes, Versatiles is a nice truly open project! One of the long term plans for OpenFreeMap is to migrate to Shortbread + Versatiles styles.
I currently use the cyclosm[1] as a tile map for my small web app. I'd love to use OpenFreeMap instead, as cyclosm is quite slow. Is there any way I can do this for bicycle maps? Awesome project.
[1] https://www.cyclosm.org
As far as I know, no vector equivalent of cyclosm has been made. OpenMapTiles's data are too poor ("compressed" : different cycle way types are summarized in a unique tag) to make a great cycle map.
What you need is a bike specific style for OpenMapTiles schema. Maybe you can find it on GitHub. Once you have the style, you can use Maputnik to tune it to your liking like on this link: https://maputnik.github.io/editor/#0.84/0/0
I found this[2] but I wouldn't know the first thing about how to integrate it, stand it up, serve it, and consume it with leaflet. I'll admit, the world of OSM is quite a labyrinth.
[2] https://github.com/cyclemap/openmaptiles-cycle
This looks really interesting. It might be a viable replacement for the back-end of an app I am working on that provides navigation for visually impaired people through spacialized sound. https://github.com/soundscape-community/soundscape
The back-end is currently based on a postgis database and a python app serving geojson tiles at zoom level 16. It currently runs on a single Hetzner server due to the resource requirements of imposm which is used to import the data into the database. Reducing hosting costs would really help us to insure the long term sustainability of the app.
I don't know yet how different the schema is and how easy it will be to support.
Soundscape was originally a Microsoft research project but they open sourced it when they shut down the app last year.
Would be interested in seeing a cost bar vs donations bar on the website.
Currently I'm paying for 3 dedicated servers at Hetzner and getting total donations of $11 USD per month.
Making this automatic might not be easy, but I might do it one day, it might help with the donations.
>getting total donations of $11
That total has gone up. :-)
Not currently having any plans of using openfreemap, but I like the cut of your jib. I'm currently in the process of generating aerial tiles for Colorado, and updating 2 of our map tile sets (bitmap tiles, Here and OpenStreetMap), but at some point we should switch to vector tiles. When we do I'll try to get a business sponsorship going, but it is nearly impossible for me to get the company to pay for sponsorship. :-(
Thank you so much for your donation, it's appreciated!
Hello. This seems like exactly something I'm looking for to save cost for my project: https://golfcourse.wiki
I'm currently using Stadia Map Tiles from when they went from free to paid.
If I'm understanding you correctly, I can switch to your instance to serve tiles to my site for free once again? I'm very happy to pitch in some cash to help keep the servers running.
I am running leaflet.js and I guess I should be able to figure out how to migrate, but if you already have a tutorial somewhere, please let me know.
This is really cool.
I've been using Google maps for a decade, and when they started offering a $200 credit, it was fine until my bill started going up to $200-300 after the credits.
It's for a hobby project that's outlasted well-funded projects, so the traffic kept increasing as competitors folded.
A few years back, I tried to run my own tileserver, but the difficulty of creating an updated vector tileset put me off, and I just continued paying the Google tax. Last month I finally got a really nasty bill, and it forced me to cancel my weekend plans and change to Mapbox.
I like Mapbox for its other API features, but I'm wondering if I could alternate between this and Mapbox, so I keep my overall costs lower. I'd prefer to donate a fixed amount instead.
Question: Does anyone have experience with switching the underlying OSM mapping services either programmatically or by webpage? I'm more asking around the end-user experience. I suppose I have to check what features of mapbox-gl 2+ I use that won't be available in the libre forks.
If you can be more specific in your questions I can probably help. I've been full time consulting in mapbox and maplibre for 7 years.
Im just someone interested in geospatial. Im a regular SWE. Seems like <1% of SWE jobs have even a nominal geospatial element like putting pins on a map.
Any tips in general? Not just for breaking in but what you wish you knew
That's an extremely general question. But mostly my take is: geospatial is not that hard. There are a lot of dinosaurs out there and legacy GIS companies who want you to believe that GIS is special and requires all kinds of specialist knowledge. But for most applications, you can just treat it as, data that happens to also have latitudes and longitudes. And then, it really is just points, lines and polygons the vast majority of the time.
Once you have specific goals about what you're trying to do with data, and how to visualise it etc, then there are usually straightforward solutions to achieving them. The difficulty can sometimes be describing them, or forming the right conceptual model.
It's worth getting really familiar with the GeoJSON spec, and using that as your data structure wherever possible.
Thanks, I mainly want to be able to load either mapbox-gl or its alternatives, then not change the rest of my code. I don't yet know how incompatible mapbox-gl 3 is with the libre options.
Best way is to try it. It just depends whether you have used any really recent features.
Just want to say thank you for making this project. I think there's a good chance I will use it for a (very low bandwidth) personal project.
Thank you very much, and also thank you for your donation!
I'll be using this this Sunday for a project for a g0v civic hackathon. You solved a problem we had been discussing today. We were going to use protomaps instead https://protomaps.com/
The pitch for the project is here if anyone in Taiwan is interested in joining: https://hackmd.io/7K4kLkseTYCPI7DWoNv5mA?view
Hackathon info: https://g0v-jothon.kktix.cc/events/g0v-hackath63n
Always on the lookout for self-hosted map servers, so I'll definitely try this one out.
Slightly off-topic, one thing I'm having a hard time finding is satellite imagery. I need to self-host an offline web application using CesiumJS, and while I can spin up a map server, I can't find satellite imagery for free or cheap. I've used MapTiler[0], their server works great and they offer low-res satellite imagery for free/testing, but their high-res images are out of my price range.
Anyone know of resources for downloading offline images of satellite imagery, compatible with OpenFreeMap or other server for use with CesiumJS? Doesn't have to be super recent images but would be nice.
[0]: https://www.maptiler.com/
Satellite images are definitely expensive, they are by far the biggest cost for MapHub. I spend more on satellite images per month than on servers. I couldn't find a high quality satellite images from free sources.
Currently researching this myself? Have you explored the satellite images from ESA/NASA? They are painful to discover but they're available to download for free and fairly (3-6 months) up to date.
I have, though admittedly not deeply enough. Agreed about the discoverability though, I never could quite find what I'm looking for (and perhaps I don't know exactly what I'm looking for)
This is amazing!
Last year I made a website for my gf where I had to build a custom map of Paris, and I struggled a lot trying to figure out how to actually make a map from scratch while avoiding paid services like mapbox.
I finally managed to hack something up using openstreetmap data, then some manual work in QGIS to customize the look, and voila - I had a bunch of folders filled with raster tiles.
This site is deployed for free on Netlify and is basically just a React SPA, a public folder with tiles, and I give the tile URL template to the OpenLayers lib to display it all nicely on the screen. Simple and it works!
I always wanted to improve the map a bit by using vector tiles as I think it looks nicer, but I thought you need a dedicated server for that? (unless I'm mistaken, correct me if I'm wrong)
Yes, you need a dedicated server with 128 GB RAM to generate the tiles. For hosting the tiles, you can use anything with 300 GB disk space, CPU is not important.
Very curious to know more about this, regarding PMTiles:
> Unfortunately, making range requests in 80 GB files just doesn't work in production. It is fine for files smaller than 500 MB, but it has terrible latency and caching issues for full planet datasets.
Wondering which part incurs the latency here.
I explained a bit about PMTiles in this comment: https://news.ycombinator.com/item?id=41636376
In summary, on a cold start (so now when it's on HN it might be totally meaningless), the following page loads in like 8-10 sec for me: https://pmtiles.io/?url=https%3A%2F%2Fdata.source.coop%2Fpro...
Technically why it's slow is entirely on CloudFlare, so I can only guess. But probably they are not mirroring files of this size across their datacenters / servers. I don't know how their public buckets hosting is done, but it's definitely very slow.
OpenFreeMap on the other hand seems almost instant for me.
See my response here: https://news.ycombinator.com/item?id=41638031
How can I use this to plot database metadata such as users, vendors, suppliers at their respective addresses. I’m developing a simple Django application and this would be a great addition.
I created a similar project, https://serverlessmaps.com which lets you host PMTiles on S3 and CloudFront, with a Lambda@Edge layer where you could also do auth...
With 1TB free traffic from the CDN, and pretty small costs for S3 and Lambda@Edge, it's probably even cheaper to self-host I guess. Even lower costs would be possible by entirely using CloudFlare services (CDN, R2)...
Replace lambda by DBOS and you'll cut down the cost by 15x, at least, and save you a lot of time developing (SAML sucks)
That's a complete other topic. DBOS also doesn't fit in this stack. And what has SAML to do with it?
I've been waiting for a good alternative to the paid map providers for a while but none of them have really hit near the level of feature parity I needed to be a good enough replacement.
As a Roman, the decision on how to prioritize the labels seems very interesting to me!
https://i.postimg.cc/tCn1xSRF/image.png
p.s. very cool project indeed
edit: I don't know how it happened, but while panning the map with the mouse, the map flipped upside down.
https://i.postimg.cc/sggZ2pHh/image.png
Thank you for sharing this, I'll make a issue about both.
The Vatican situation is trickier than you'd think, as it's on country level, while Rome is on city level, and naturally you'd want to give countries a priority over cities. Like on low zooms you only want to show countries, then as it gets closer you show cities. Well, except for Vatican.
Nearby San Marino needs similar treatment.
Looks great love the free tiles and no limits or API keys.
Question, why use btrfs?
So a full planet OSM extract is about 300 million binary files, in total about 90 GB. The most popular ways to store it:
- MBTiles - an SQLite file, each file is a row in a table, you need a server to serve it.
- PMTiles - a single file, optimised for serverless usage.
- Extract them into a directory, which in practice should be on a partition image. This is the approach I choose.
I tested ext4 and Btrfs and I choose Btrfs. The reason is how ext4 and Btrfs handles inodes. Btrfs handles inodes so much better compared to ext4, it doesn't allocate them at start and also allows putting tiny files right with the metadata.
Because here the average tile size is only 405 bytes, most of the tiles can actually stored with the Btrfs metadata block. From the latest run:
Btrfs data is 51.57GiB
Btrfs metadata is 84.37GiB
This sounds like a perfect application for EROFS[1]. While it comes from an embedded systems background, it has seen some usage in container use cases and is moving towards a general "mountable tar" application. It would also avoid the tedium you have to go through in shrink_btrfs.py because you can just generate the image out of a tree.
I wanted to give repackaging the btrfs image a shot but the download was pretty slow - I assume your server is getting HN-hugged a bit so I didn't want to make it worse and stopped the download.
[1] https://erofs.docs.kernel.org/en/latest/index.html
Thanks a lot, I didn't know about it! I also liked the fact that Btrfs is probably super well tested in the Linux kernel by now.
btrfs.openfreemap.com just a public Cloudflare bucket, no idea why it might be slow.
> I also liked the fact that Btrfs is probably super well tested in the Linux kernel by now.
btrfs has certainly been around for longer, but in my (embedded systems only) experience, EROFS has been pretty solid - it's slowly being picked up by Android, so it is definitely seeing a lot of use in the wild (probably surpassing btrfs by the number of installations already).
> btrfs.openfreemap.com just a public Cloudflare bucket, no idea why it might be slow.
I'm getting 30 MiB/s (on a gigabit uplink) - not great, not terrible. A .torrent would be nice but I guess outside of being on the HN front page full-planet downloads by different people won't synchronize enough for this to be useful (and using web seeds is problamtic in its own right with small-ish chunks).
Do you think btfs would work well for old school pre rendered raster tiles on disk?
Yes, I think it would. Have a look at the extract_mbtiles script: https://github.com/hyperknot/openfreemap/blob/main/modules/t...
I apologize if this is something stupid, or answered elsewhere, but is there a good way to trim Planet down to a smaller area?
I've been making some custom map stuff for mountain bike trails and I'd really like to move to self-hosted vector tiles for all layers, but too much of what I find says to start with Planet.osm when all I really need is a State (in the US) or even a few-miles-wide area.
(My goal is to basically snapshot OSM data, generate tiles, and use that until I decide to do another snapshot down the line, so the underlying data doesn't change. And limit it to a small area because that's all I need.)
Examples of maps I've done this way, and want to improve, are: https://trailmaps.app/ramba/ and https://trailmaps.app/dte/
Look into Planetiler [1] (which OP uses for tile generation). It supports downloading regions that are listed on Geofabrik [2] and converting them to mbtiles or pmtiles. Geofabrik has separate extracts for all US states, for example. If you need to extract an even smaller area from that result, GDAL has support for mbtiles so you could use gdalwarp [3] to extract a new mbtiles file out of it using bounds.
Another option is to use the extract functionality in pmtiles [4] to extract your area of interest from their daily full-planet pmtiles builds. You can then statically host that file and use that in your client with one of their client libraries.
[1] https://github.com/onthegomap/planetiler
[2] https://download.geofabrik.de/
[3] https://gdal.org/en/latest/programs/gdalwarp.html
[4] https://docs.protomaps.com/pmtiles/cli#extract
If you know the tile numbers, you can just copy them out of the Btrfs image.
Afaik btrfs supports incremental send / sync very efficiently, so if you just want an update it saves bandwidth
Right on time! Thank you and congratulations for your release. I was looking for gmaps and mapbox alternatives this morning and right now I'm considering this.
Very nice!
Curious that you describe the OSM Bright style as "abandoned by their upstream project". I see edits 4 months old (https://github.com/openmaptiles/osm-bright-gl-style), but I only looked superficially. Is it really abandoned?
Since I wrote it, one person made 1 commit related to the visual style. Before that, 1 visual commit in 2023 and then 2021 is when work kind of stopped.
Thank you so much for sharing the setup and the hosted service.
I'm a little curious about the setup bit for (self) hosting - it's essentially a series of bespoke python scripts? Not cloud init, not Ansible - and not shell scripts - nor Terraform/tofu.
Would love to hear a few thoughts of how you arrived at this setup - and if you re-use parts for other projects?
Thanks! It's not so bespoke, it's Fabric (https://www.fabfile.org/), it's been around at least since 2011, but probably much longer, since they already had 1.2.0 in 2011. It's a super reliable script for setting up servers over SSH. Yes, you run the same commands you'd type in a terminal, works perfectly.
Thanks for clearing that up - it wasn't obvious to me at a glance.
Not OP but I do the same thing, except with NodeJS.
I spent a long time learning Salt stack (like Ansible) but it never really paid off. When you're writing o config to set up one specific project on known infrastructure you might as well just code it in the scripting language you like best.
I miss some libre client for Google's Street View. On 3D maps, adapting Marble for OSGEarth should be a piece of cake.
If not, declaring one with IMGui+layers can be done in a week from any experienced C++ programmer. The included demos for OSGEarth already depict a minimal client with impressive 3D maps for a nearly non-supported demo.
I think KDE/Plasma lost a great oportunity there by not adopting OSGEarth.
What would be the aim of a libre client for Google's SW ? Street View API is quite expensive.
Do you know about https://panoramax.fr ?
I'm relatively new to mapping, can you explain the difference between running Leaflet and a free map tiling provider compared to your product?
As far as I know, there are no "Free Map Tiling provider"-s. This project aims to be exactly one. You can use it with Leaflet using the https://github.com/maplibre/maplibre-gl-leaflet plugin.
I asked the question as a top level comment but also duplicating here. Isn't Apache Baremaps a "free map tiling provider"? Or do I misunderstand what a tiling provider is?
https://baremaps.apache.org/
They are providing the building blocks for making a self-hosted vector tiles server, they don't actually host the tiles.
Interesting thanks, I've used map tilers in the past without paying, maybe I just never came up against any rate limiting
Thanks for this free service Zsolt. I hope it's not abused (too much traffic) and that donations from real users fund this very good initiative ;-)
On a related subject, I remember seeing MapHub sometime ago and I have it in my bookmarks for one of my (forgotten?) projects. Whenever I find some free time I need to sit down and try the free tier to see if it will do what I have in mind.
Thanks a lot for your words! I'm also curious whether the donations will be able to cover the bandwidth costs. Currently I'm paying for 3 dedicated servers at Hetzner and getting a total of $11 per month from donations.
Hey, just curious. If you have no Cloud/LB/CDN how are you routing requests to datacenters? Anycast? DNS? Something else?
Round Robin DNS. The browser selects the best server, based on ping or some other internal metric.
Also thanks for the donation!
Seems like a great service! We could definitely consider this as an viable option for our startup's application. I am not an expert regarding the OSM data and maps in general, but how customizable is your library, can I somehow relatively easy add housenumbers of the addresses from OSM data on the buildings in the map?
This is totally customizable and the house numbers are already present in the original data, it's just not displayed in the default styles. You can customize the style by using the Maputnik editor, house numbers are visible in the "Inspector" view. https://maputnik.github.io/editor?style=https://tiles.openfr...
Very cool, love the various tile sets (especially the 3d layer!)
One thing that tripped me up is the zoom -- the affordance for zoom in/out across most other mapping UIs is via mouse motions -- how come in your demo it's restricted to the +/- buttons? Perhaps I missed something...
Thanks for sharing this, looks very neat.
That's something that most tile viewers let you customize in the code. Sometimes you want to disable scroll wheel zoom so as to not interrupt page scrolling, like when the map takes up a large portion of the width of the page.
On the main page I disabled scroll zoom, because it hijacks your scroll and literally you cannot scroll through the map. It makes sense for full-screen maps only.
Of course, you have 100% control over exactly what controls you configure with MapLibre, when you set up your app.
Ah nice, of course it's configurable :)
I know the github says docker free.... What would it take to dockerize so people can self host, say on kubernetes.
Someone making a Docker image from this. Instead of using Fabric's commands like c.sudo('git clone ...') it'd use the RUN commands.
I think it would be quite a straightforward task to transform it into a Docker file. I'd be happy to link to it on GitHub.
About displaying the vector tiles, which project goes well with it? I like Leaflet and worked extensively with it, but it has some compatibility problems with vector tiles and other small issues when dragging vector tiles for example. Is Maplibre the gold standard for vector tiles now?
Maplibre works fine, there is even a plugin so you can use it in Leaflet. And it's a well maintained project.
The other one is Mapbox GL from which Maplibre was forked. But you only use that with their tiles now.
Yes, MapLibre is the gold standard now.
Yes. It's active. But it's imperative. I didn't find a good React interface to MapLibre.
See Streetcomplete switching to MapLibre.
https://visgl.github.io/react-map-gl/ is a react interface to MapLibre
I am currently exploring the open source mapping world and trying to get a better grasp of what's available.
How does this project compare to Apache Baremaps (incubating)?
https://baremaps.apache.org/
They are providing the building blocks for making a self-hosted vector tiles server, they don't actually host the tiles.
Hard links are to deduplicate identical tiles, right? How much does that save and how close is the tile with most links (empty one(s)?) to the filesystem limits? Inquiring minds want to know...
Yes, exactly, it's to save space. For example there is one such tile for "ocean", which covers 70% of earth.
In total, there are 271 million hard links. So out of 300 million files 271 are hard links!
The file system limit is 64k hard-links for the same file, so I have to handle the case when it's reached and then start a new file for the next 64k.
> The file system limit is 64k hard-links for the same file
I had never heard that, so I went sniffing around and it seems to be ext4 specific[1] but I wasn't able to easily get the limits for ZFS (or xfs, etc), so depending on how much glucose one wished to spend it may be better to use a different FS than all that renaming work around
1: https://en.wikipedia.org/wiki/Hard_link#Limitations
The whole extract_mbtiles.py file is 97 lines of code. This contains parsing MBTiles, writing metadata and some CLI specific lines. It's actually quite a concise script for doing this while taking care of the hard-link limits.
Just out of curiosity, is there no clever way to solve this within nginx with rules instead?
In nutiteq mobile maps SDK (later Carto, now abandonware) we used specifically compressed bitmap to represent 'water' and 'empty land' tilemasks to cover these two special cases. We provided planet-scale mobile embedded mbtiles package in 30GB if I remember well. This tile mask (quite instant bitmap index) concept should work well for server case also.
The Linux kernel's filesystem cache is actually really efficient at doing this. I doubt we could come up with a nginx scripting solution which could be equally efficient.
Awesome. I've been waiting forever for a vector OSM replacement. I already changed it to be the default for my open source GPS tracking project.
Seems like the full world download links dont work, 404: https://btrfs.openfreemap.com/
CloudFlare buckets don't support indexing / directory listing, so you need to get the exact URL. To help with this, here is a file listing the content of the bucket: https://btrfs.openfreemap.com/files.txt
Very good initiative, I will try it out for my project and donate monthly if it is successful!
Awesome! Moreover, I love how hyperknot is engaging with the people and the feedback :) This is super inspiring.
This is fantastic, thank you!
Great project! Do you support geocoding with a nominatim api?
I would be interested in the differences between this and protomaps
I'll write a detailed blog post for that, definitely.
Basically Protomaps / PMTiles allows you to do this serverless, but it has it's downsides.
There are two ways to use PMTiles with Cloudflare:
- Putting the file in a public bucket and use HTTP range requests.
- Deploy a worker to access it.
For the range request version, you can test PMTiles here: https://pmtiles.io/?url=https%3A%2F%2Fdata.source.coop%2Fpro....
It loads in 8-10 sec for me on a cold start. Of course it gets faster if many of us try to request it at the same time, but if you check it once per day for example, it'll be really slow, up to 10 sec.
Compare that with OpenFreeMap here: https://openfreemap.org/quick_start
It's near instant for me. (I know it's not full screen but you can make the app and then see how it's in full screen).
For the Cloudflare Workers version, I didn't find any publicly available full planet test, so I don't know the performance, but Workers are not free. It might not matter for a small project, but the $0.30/million requests per month can easily add up if your project gets popular.
In terms of OpenFreeMap, hosting it on a Cloudflare Worker would be prohibitively expensive.
I'm the developer of Protomaps, to summarize:
The latency you see on https://pmtiles.io/?url=https%3A%2F%2Fdata.source.coop%2Fpro... is representative of how PMTiles works on AWS S3, coming from the us-west-2 region. It will be reasonable to load for those in the western US and likely quite slow from Europe or Oceania.
If you want to make a direct comparison of Protomaps to OpenFreeMap, you need to compare serving OpenFreemap with NGINX from btrfs on disk, to running `pmtiles serve` on a `.pmtiles` file on disk, as described here: https://docs.protomaps.com/deploy/server
The OpenFreeMap page for me (in Taiwan) takes 1-2 seconds per tile, which is more than double the load tile for the PMTiles in us-west-2 example linked above.
The best solution to get latency competitive with commercial providers, for a global audience, is to cache tiles at CDN edge locations. I describe automated ways to do that with Cloudflare Workers and AWS Lambda + Cloudfront here:
https://docs.protomaps.com/deploy/cloudflare https://docs.protomaps.com/deploy/aws
I'm also experimenting with http://tigrisdata.com which lets you geo-distribute a static storage bucket like in this example: https://pmtiles.io/?url=https%3A%2F%2Fdemo-bucket.protomaps....
No, the ping is 150 ms to us-west-2, and the tiles load in like 5 seconds on a cold start. Of course we cannot test cold start on HN comments because HN is the definition of hot :-)
I can imagine workers to be fast, it's the range requests which are super slow. It's also outside of your control, it depends on how Cloudflare and S3 handles range requests to 90 GB files.
I think if you could make PMTiles split into files <10 MB, it'd be perfect with range requests.
I agree, there are tradeoffs to using static storage - the intended audience for PMTiles is those that prefer using static sites instead of administering a Linux server.
I would be interested to see a comparison of Btrfs + nginx serving latency, vs `pmtiles serve` from https://github.com/protomaps/go-pmtiles on a PMTiles archive on disk. That would be a more direct comparison.
I think there's potentially some interesting use case for tiles in Btrfs volumes and incremental updates, which I haven't tackled in PMTiles yet!
I think both solutions could easily saturate a 1 Gbps line. I benchmarked Btrfs + nginx and it could do 30 Gbps, which doesn't really make a difference if your server is 1 Gbps only.
The fact that there is no service running was the more important for me. Mostly for security and bugs. I had so many problems with various tile servers in production, they needed daily restarting, they had memory leaks, etc.
Basically I wanted to go nginx-only for security and to avoid tile server bugs.
I see, I think that's a good approach to enable serving with stock nginx as well as for companies that are built on Nginx or a plain HTTP serving stack already.
For PMTiles the module is loadable directly as a Caddy plugin (https://docs.protomaps.com/deploy/server#caddyfile) which I prefer to nginx for security and bugs (and automatic SSL), and also enables serving PMTiles from disk or a remote storage bucket without a separate service running.
Yes, PMTiles with the Caddy plugin is very similar to nginx + Btrfs.
At that point, the difference between the two projects is mostly which schema is being used.
You both specify the filesystem to be Btrfs. Is there any advantage in this case against ZFS, ext4, XFS... or is it just a practical choice?
Yes, small files can fit in the metadata, which makes a super big difference when you have 300 million files of 405 bytes each. Also, inode handling is way better compared to ext4.
Oh neat! Thanks! I'm checking the docs and I guess you're referring to "Inline files". I don't have much knowledge about btrfs, so I didn't know...
The nearest thing (and it's not really really similar, just related) that ZFS has is special VDEVs.
When you have an array of disks (usually "slow", like regular HDDs) you can attach to it another array of disks (usually very fast, like NVMe) where you can store metadata (file information) and optionally small files up to a size that you can define.
So for example you have a 50TB (or 500, who knows) array of SATA disks, and a small but superfast array of NVMe drives. Lets say 128GB. Or 512, or 1TB depending on you want to do.
File metadata is saved there, so doing a find, ls, tree... operation is now very fast. And if you save, for example, all files smaller than 32KB there (it will depend on your needs, also) all the small file operations will be way faster.
Yes, the key thing about Btrfs is how it handles inodes and how it can store data with the metadata. In the OpenFreeMap image, 60% of the files are stored with the metadata, essentially taking up no space.
I would also like to see this comparison. And for good measure it’d be great to also include Versatiles in this comparison.
If you would like to run this comparison for tileservers reading from disk, I wrote a small tool to simulate traffic based on access logs of OpenStreetMap tiles:
https://github.com/bdon/TileSiege
Anybody know of a similar service for weather information?
US-only, I've had good luck with the NWS JSON API[1]. Someone posted a while ago on here[2] about taking in various NWS models and creating "Pirate Weather"[3], a drop-in replacement for Dark Sky.
1. https://www.weather.gov/documentation/services-web-api
2. https://news.ycombinator.com/item?id=34329988
3. https://pirateweather.net/
How much disk space do you need to host it?
I'd recommend at least 200 GB.
Thanks! That's a lot less than I expected.
Very interesting, I might try this at $WORK.
leafletjs and openstreetmap is also cool
[flagged]