Compare commits

...

17 Commits
oam ... osmbe

Author SHA1 Message Date
nobohan
85b7a8261d Add a lang parameter to the run script 2025-05-02 15:06:14 +02:00
nobohan
4efa0b0ea1 add nl and fr map style in renderd.conf 2025-05-02 14:49:44 +02:00
nobohan
64b3940c5c upd branch for osmbe map style 2025-05-02 14:49:17 +02:00
nobohan
96d4fe815d upd README for osmbe tile generation 2024-04-13 08:51:13 +02:00
nobohan
917337acc6 osmbe: render also low zoom levels 2024-04-03 13:58:05 +02:00
nobohan
a10fd7a309 set work_mem to 250MB in postgres (in run.sh) 2024-04-02 21:25:09 +02:00
nobohan
7c2044be23 Add osmium command and tiles.list generation 2024-04-02 21:18:58 +02:00
nobohan
bcb3823ef5 tile list for buffered belgium 2024-03-26 10:40:16 +01:00
nobohan
0524d471f4 update README and run.sh 2024-03-18 16:01:44 +01:00
nobohan
8c83b3adb7 add BE coordinates in generate-tiles-list 2024-02-26 09:32:42 +01:00
nobohan
b7623037c4 add external-data yml for osmbe 2024-02-25 18:48:43 +01:00
nobohan
9d1b824e0f add scripts for osmbe and run it 2024-02-25 18:27:31 +01:00
nobohan
fbb68e8c4f add symbols for osmbe style 2024-02-25 18:01:17 +01:00
nobohan
b714947ce1 correct paths for style and lua files in import command 2024-02-25 11:20:37 +01:00
nobohan
c029d55383 upd README 2024-02-25 11:18:10 +01:00
nobohan
92ef9c3f2e osm-be: adapt repo and path + import command 2024-02-25 11:14:46 +01:00
nobohan
d49c6be309 change paths and files for osmbe map style 2024-02-09 18:46:05 +01:00
12 changed files with 272521 additions and 46 deletions

1
.gitignore vendored
View File

@@ -1,5 +1,6 @@
pbf/*.pbf
pbf/*.poly
!pbf/polyBEbuffer.poly
pbf/shp/*
pbf/*.tif
tiles/*

View File

@@ -5,42 +5,29 @@
This is an adaptation of https://github.com/Overv/openstreetmap-tile-server/ for testing different cartoCss styles and generating tiles.
### Use with docker-compose
### Use with docker compose
#### Fresh install
```bash
#docker volume create osm-data
docker-compose build
docker compose build
```
For entering into the container, use the "terminal" argument of the run.sh script:
```bash
docker-compose run --rm map terminal
docker compose run --rm map terminal
```
#### OAM data dependencies
##### Contour lines
Download the geopackage from http://opendata.champs-libres.be/beautiful-contour-belgium.zip, convert it to shp and place it into the pbf/shp folder under the name "contours.shp". It will be mounted as written in the `docker-compose.yml` file.
The shp is imported using `shp2pgsql` in the import step, assuming EPSG:31370 as the input shp CRS.
#### Hillshade
Download the hillshade at http://opendata.champs-libres.be/hillshade_belgium_EPSG3857.zip and place the tif file in the pdf folder, as mounted in the `docker-compose.yml` file.
#### Do an import
Download a pbf and a poly file of your choice on https://download.geofabrik.de and put it in the `pbf` folder. Change the lines of the pbf and poly paths in the volumes in the `docker-compose.yml` file
Download a pbf and a poly file of your choice on https://download.geofabrik.de and put it in the `pbf` folder. Change the lines of the pbf and poly paths in the volumes in the `docker compose.yml` file
Then, you can import the OSM data by doing this command:
```bash
docker-compose run --rm map import
docker compose run --rm map import
```
This will take a while, about 15 min for Luxembourg for example.
@@ -60,7 +47,7 @@ INFO:root: Import complete
#### Run the server to produce and view tiles
```bash
docker-compose run --rm map run
docker compose run --rm map run
```
The tiles are available on http://192.168.176.2/ or something like that. Have a look at the docker container to know which is the IP address: there is a message like "Could not reliably determine the server's fully qualified domain name, using 192.168.176.2.". (should be on http://localhost:8080/ though...)
@@ -76,11 +63,13 @@ python3 generate-tiles-list.py > tiles.list
Move this `tiles.list` file in the `/tiles/` directory and then run:
```bash
docker-compose run --rm map generate_tiles
docker compose run --rm map generate_tiles
```
For Belgium up to zoom 18, it might take >48 hours!
NB: use `cat tiles.list | grep " 18" > tiles_18.list` to split the tiles.list per zoom level
#### Transfer the tiles to tiles.champs-libres.be
@@ -88,6 +77,32 @@ For Belgium up to zoom 18, it might take >48 hours!
rsync --compress --recursive ./tiles/cyclosm debian@juniperus.champs-libres.be:/var/lib/mod_tile/cyclosm
```
#### Things to do for the osmbe style
##### Clip a europe.pbf file with osmium
We want a buffer around Belgium with some generated tiles. To do so, we clip a larger pbf file like `europe.osm.pbf` with a poly file, which can be generated with http://polygons.openstreetmap.fr/get_poly.py?id=52411&params=0.150000-0.005000-0.005000
Then, we clip the europe pbf with osmium:
```bash
osmium extract --strategy smart --polygon=polyBEbuffer.poly --output=belgium-buffer.osm.pbf --overwrite /mnt/tera/Projets/OSM/Data/europe-latest.osm.pbf
```
The `poly` file is kept in the pbf folder.
##### Generate a tiles.list file with the buffer around Belgium
This file was generated with the script `utils/generate-tiles-list-buffer.py`. The result is also kept in this repo in `utils/tiles-BE-buffer.list`.
#### Troubleshooting
##### Role rendered already exists
While importing, if you encounter the error "createuser: error: creation of new role failed: ERROR: role "renderer" already exists", it is because the db already exist from a previous import, just remove/rename the `postgres` folder into database.
## Original README
[![Build Status](https://travis-ci.org/Overv/openstreetmap-tile-server.svg?branch=master)](https://travis-ci.org/Overv/openstreetmap-tile-server) [![](https://images.microbadger.com/badges/image/overv/openstreetmap-tile-server.svg)](https://microbadger.com/images/overv/openstreetmap-tile-server "openstreetmap-tile-server")
@@ -191,7 +206,7 @@ Your tiles will now be available at `http://localhost:8080/tile/{z}/{x}/{y}.png`
### Using Docker Compose
The `docker-compose.yml` file included with this repository shows how the aforementioned command can be used with Docker Compose to run your server.
The `docker compose.yml` file included with this repository shows how the aforementioned command can be used with Docker Compose to run your server.
### Preserving rendered tiles

View File

@@ -7,9 +7,9 @@ services:
#- osm-data:/data/database/
- ./database:/data/database/
- ./tiles:/data/tiles/
- ./oam/renderd.conf:/etc/renderd.conf
- ./osmbe/renderd.conf:/etc/renderd.conf
- ./run.sh:/run.sh # for dev
- ./pbf/luxembourg-latest.osm.pbf:/data/region.osm.pbf # change here your pbf
- ./pbf/belgium-buffer.osm.pbf:/data/region.osm.pbf # change here your pbf
- ./pbf/luxembourg.poly:/data/region.poly # and your poly
- ./pbf/shp/:/data/shp/
- ./pbf/hillshade.tif:/data/hillshade.tif

View File

@@ -12,10 +12,26 @@ font_dir=/usr/share/fonts
font_dir_recurse=true
; ADD YOUR LAYERS:
[oam]
[osmbe]
URI=/tile/
TILEDIR=/var/cache/renderd/tiles
XML=/data/style/mapnik.xml
HOST=localhost
TILESIZE=256
MAXZOOM=20
[osmbe_nl]
URI=/tile_nl/
TILEDIR=/var/cache/renderd/tiles
XML=/data/style/mapnik-nl.xml
HOST=localhost
TILESIZE=256
MAXZOOM=20
[osmbe_fr]
URI=/tile_fr/
TILEDIR=/var/cache/renderd/tiles
XML=/data/style/mapnik-fr.xml
HOST=localhost
TILESIZE=256
MAXZOOM=20

1
pbf/osmium_extract.sh Normal file
View File

@@ -0,0 +1 @@
osmium extract --strategy smart --polygon=polyBEbuffer.poly --output=belgium-buffer.osm.pbf --overwrite europe-latest.osm.pbf

196
pbf/polyBEbuffer.poly Normal file
View File

@@ -0,0 +1,196 @@
polygon
1
2.275 51.17
2.245 51.22
2.24 51.28
2.255 51.335
2.29 51.385
2.53 51.48
2.565 51.485
2.68 51.545
2.8 51.585
2.965 51.675
3.06 51.7
3.15 51.685
3.275 51.62
3.35 51.59
3.395 51.56
3.505 51.435
3.555 51.45
3.61 51.455
3.74 51.42
3.795 51.415
3.915 51.37
3.93 51.37
3.97 51.39
4.03 51.395
4.055 51.405
4.075 51.415
4.09 51.455
4.11 51.48
4.155 51.51
4.21 51.525
4.25 51.525
4.28 51.56
4.325 51.59
4.46 51.63
4.585 51.625
4.63 51.6
4.64 51.6
4.685 51.635
4.745 51.655
4.8 51.655
4.84 51.645
4.885 51.63
4.925 51.605
4.94 51.605
4.995 51.635
5.085 51.63
5.16 51.6
5.185 51.58
5.225 51.525
5.25 51.465
5.33 51.425
5.385 51.42
5.435 51.445
5.465 51.45
5.56 51.44
5.61 51.415
5.62 51.4
5.655 51.38
5.685 51.345
5.71 51.335
5.73 51.34
5.79 51.335
5.835 51.315
5.9 51.3
5.965 51.25
5.985 51.225
6.005 51.165
6 51.105
5.965 51.025
5.915 50.965
5.905 50.92
5.94 50.91
6.045 50.905
6.09 50.89
6.11 50.875
6.18 50.86
6.205 50.845
6.24 50.805
6.315 50.785
6.355 50.765
6.39 50.73
6.41 50.695
6.425 50.62
6.465 50.585
6.5 50.54
6.515 50.505
6.525 50.48
6.525 50.425
6.545 50.4
6.56 50.345
6.555 50.295
6.545 50.26
6.505 50.21
6.43 50.17
6.375 50.155
6.33 50.125
6.31 50.09
6.225 50.005
6.16 49.98
6.11 49.98
6.04 50.005
6.01 49.99
5.975 49.915
5.935 49.86
5.98 49.83
6.02 49.78
6.055 49.705
6.06 49.65
6.05 49.595
6.015 49.52
5.985 49.475
5.95 49.445
5.875 49.405
5.81 49.395
5.8 49.4
5.77 49.39
5.705 49.39
5.68 49.37
5.65 49.36
5.59 49.355
5.55 49.365
5.515 49.35
5.475 49.345
5.435 49.355
5.435 49.35
5.365 49.39
5.33 49.425
5.31 49.46
5.265 49.47
5.21 49.5
5.18 49.545
5.135 49.545
5.085 49.565
5.04 49.59
5.005 49.625
4.965 49.64
4.94 49.635
4.83 49.64
4.76 49.675
4.72 49.72
4.71 49.745
4.7 49.805
4.705 49.82
4.695 49.845
4.64 49.84
4.565 49.805
4.425 49.79
4.295 49.815
4.18 49.805
4.15 49.81
4.06 49.85
4.035 49.87
3.995 49.945
3.985 49.995
3.995 50.065
3.975 50.145
3.99 50.19
3.9 50.175
3.82 50.19
3.745 50.155
3.705 50.155
3.64 50.17
3.605 50.19
3.555 50.24
3.51 50.34
3.505 50.335
3.47 50.34
3.445 50.35
3.365 50.34
3.22 50.39
3.18 50.42
3.145 50.47
3.125 50.55
3.105 50.585
3.1 50.61
3.08 50.625
3.05 50.62
3.01 50.58
2.97 50.555
2.94 50.545
2.87 50.545
2.745 50.58
2.69 50.61
2.645 50.665
2.615 50.665
2.56 50.68
2.485 50.75
2.455 50.8
2.44 50.94
2.43 50.955
2.42 51.005
2.275 51.17
END
END

51
run.sh
View File

@@ -13,21 +13,21 @@ function setPostgresPassword() {
}
function compileStyle() {
local lang=$1
cd /data/style/
git clone --single-branch --branch osm-tile-server https://github.com/nobohan/OpenArdenneMap.git --depth 1
# mv symbols folder
mkdir /data/img/
mv /data/style/OpenArdenneMap/img/* /data/img/
# copy fonts
cp /data/style/OpenArdenneMap/fonts/* /usr/share/fonts/
# change dbname
sed -i 's/"dbname": "osmpg_db",/"dbname": "gis",/g' ./OpenArdenneMap/osm2pgsql/project.mml
# change hillshade
sed -i 's,/mnt/tera/ChampsLibres/Projets/OSM/CyclOSM/hillshade/data/dem_be_10x10_hillshade_semi_transparent_3857.tif,/data/hillshade.tif,g' ./OpenArdenneMap/osm2pgsql/project.mml
# change contours
sed -i 's,../contour/beautiful_contour_belgium.shp,/data/shp/contours_3857.shp,g' ./OpenArdenneMap/osm2pgsql/project.mml
# carto build
carto OpenArdenneMap/osm2pgsql/project.mml > mapnik.xml
git clone --single-branch --branch update-2025 https://github.com/nobohan/openstreetmap-carto-be --depth 1
mkdir /data/style/patterns/
mv /data/style/openstreetmap-carto-be/patterns/* /data/style/patterns/
mkdir /data/style/symbols/
mv /data/style/openstreetmap-carto-be/symbols/* /data/style/symbols/
mkdir /data/style/scripts/
mv /data/style/openstreetmap-carto-be/scripts/* /data/style/scripts/
mv /data/style/openstreetmap-carto-be/external-data.yml /data/style/external-data.yml
if [ -z "$lang" ]; then
carto openstreetmap-carto-be/project.mml > mapnik.xml
return 1
fi
carto "openstreetmap-carto-be/project-${lang}.mml" > mapnik-${lang}.xml
}
@@ -54,7 +54,7 @@ fi
if [ "$1" == "import" ]; then
compileStyle
compileStyle $2
# Ensure that database directory is in right state
mkdir -p /data/database/postgres/
@@ -71,9 +71,9 @@ if [ "$1" == "import" ]; then
sudo -u postgres createdb -E UTF8 -O renderer gis
sudo -u postgres psql -d gis -c "CREATE EXTENSION postgis;"
sudo -u postgres psql -d gis -c "CREATE EXTENSION hstore;"
sudo -u postgres psql -d gis -c "CREATE EXTENSION postgis_sfcgal;"
sudo -u postgres psql -d gis -c "ALTER TABLE geometry_columns OWNER TO renderer;"
sudo -u postgres psql -d gis -c "ALTER TABLE spatial_ref_sys OWNER TO renderer;"
sudo -u postgres psql -d gis -c "SET work_mem TO '250MB';"
setPostgresPassword
# Download Luxembourg as sample if no data is provided
@@ -112,7 +112,7 @@ if [ "$1" == "import" ]; then
fi
# Import data
sudo -u renderer osm2pgsql -d gis --create --slim -G -S /data/style/OpenArdenneMap/osm2pgsql/OpenArdenneMap.style --extra-attributes \
sudo -u renderer osm2pgsql -d gis --create --hstore -G -S openstreetmap-carto-be/openstreetmap-carto.style --tag-transform-script openstreetmap-carto-be/openstreetmap-carto.lua \
/data/region.osm.pbf \
${OSM2PGSQL_EXTRA_ARGS:-} \
;
@@ -128,7 +128,11 @@ if [ "$1" == "import" ]; then
sudo -u postgres psql -d gis -f /data/style/${NAME_SQL:-indexes.sql}
fi
sudo -u postgres psql -d gis -f /data/style/OpenArdenneMap/create_views.sql
#Import external data
chown -R renderer: /home/renderer/src/ /data/style/
if [ -f /data/style/scripts/get-external-data.py ] && [ -f /data/style/external-data.yml ]; then
sudo -E -u renderer python3 /data/style/scripts/get-external-data.py -c /data/style/external-data.yml -D /data/style/data
fi
# Register that data has changed for mod_tile caching purposes
sudo -u renderer touch /data/database/planet-import-complete
@@ -142,7 +146,7 @@ if [ "$1" == "run" ]; then
# Clean /tmp
rm -rf /tmp/*
compileStyle
compileStyle $2
# migrate old files
if [ -f /data/database/PG_VERSION ] && ! [ -d /data/database/postgres/ ]; then
@@ -208,15 +212,20 @@ fi
if [ "$1" == "generate_tiles" ]; then
compileStyle
compileStyle $2
service postgresql start
service apache2 restart
sudo -u renderer renderd -c /etc/renderd.conf
cat /data/tiles/oam/tiles.list | render_list --map oam
if [ -z "$2" ]; then
render_list --map osmbe -n 36 --min-zoom=0 --max-zoom=7 --all
cat /data/tiles/osmbe/tiles.list | render_list --map osmbe -n 36
bash
render_list --map osmbe_${$2} -n 36 --min-zoom=0 --max-zoom=7 --all
cat /data/tiles/osmbe/tiles.list | render_list --map osmbe_${$2} -n 36
bash
fi

View File

@@ -0,0 +1 @@
{"type":"Polygon","coordinates":[[[2.275,51.17],[2.245,51.22],[2.24,51.28],[2.255,51.335],[2.29,51.385],[2.53,51.48],[2.565,51.485],[2.68,51.545],[2.8,51.585],[2.965,51.675],[3.06,51.7],[3.15,51.685],[3.275,51.62],[3.35,51.59],[3.395,51.56],[3.505,51.435],[3.555,51.45],[3.61,51.455],[3.74,51.42],[3.795,51.415],[3.915,51.37],[3.93,51.37],[3.97,51.39],[4.03,51.395],[4.055,51.405],[4.075,51.415],[4.09,51.455],[4.11,51.48],[4.155,51.51],[4.21,51.525],[4.25,51.525],[4.28,51.56],[4.325,51.59],[4.46,51.63],[4.585,51.625],[4.63,51.6],[4.64,51.6],[4.685,51.635],[4.745,51.655],[4.8,51.655],[4.84,51.645],[4.885,51.63],[4.925,51.605],[4.94,51.605],[4.995,51.635],[5.085,51.63],[5.16,51.6],[5.185,51.58],[5.225,51.525],[5.25,51.465],[5.33,51.425],[5.385,51.42],[5.435,51.445],[5.465,51.45],[5.56,51.44],[5.61,51.415],[5.62,51.4],[5.655,51.38],[5.685,51.345],[5.71,51.335],[5.73,51.34],[5.79,51.335],[5.835,51.315],[5.9,51.3],[5.965,51.25],[5.985,51.225],[6.005,51.165],[6,51.105],[5.965,51.025],[5.915,50.965],[5.905,50.92],[5.94,50.91],[6.045,50.905],[6.09,50.89],[6.11,50.875],[6.18,50.86],[6.205,50.845],[6.24,50.805],[6.315,50.785],[6.355,50.765],[6.39,50.73],[6.41,50.695],[6.425,50.62],[6.465,50.585],[6.5,50.54],[6.515,50.505],[6.525,50.48],[6.525,50.425],[6.545,50.4],[6.56,50.345],[6.555,50.295],[6.545,50.26],[6.505,50.21],[6.43,50.17],[6.375,50.155],[6.33,50.125],[6.31,50.09],[6.225,50.005],[6.16,49.98],[6.11,49.98],[6.04,50.005],[6.01,49.99],[5.975,49.915],[5.935,49.86],[5.98,49.83],[6.02,49.78],[6.055,49.705],[6.06,49.65],[6.05,49.595],[6.015,49.52],[5.985,49.475],[5.95,49.445],[5.875,49.405],[5.81,49.395],[5.8,49.4],[5.77,49.39],[5.705,49.39],[5.68,49.37],[5.65,49.36],[5.59,49.355],[5.55,49.365],[5.515,49.35],[5.475,49.345],[5.435,49.355],[5.435,49.35],[5.365,49.39],[5.33,49.425],[5.31,49.46],[5.265,49.47],[5.21,49.5],[5.18,49.545],[5.135,49.545],[5.085,49.565],[5.04,49.59],[5.005,49.625],[4.965,49.64],[4.94,49.635],[4.83,49.64],[4.76,49.675],[4.72,49.72],[4.71,49.745],[4.7,49.805],[4.705,49.82],[4.695,49.845],[4.64,49.84],[4.565,49.805],[4.425,49.79],[4.295,49.815],[4.18,49.805],[4.15,49.81],[4.06,49.85],[4.035,49.87],[3.995,49.945],[3.985,49.995],[3.995,50.065],[3.975,50.145],[3.99,50.19],[3.9,50.175],[3.82,50.19],[3.745,50.155],[3.705,50.155],[3.64,50.17],[3.605,50.19],[3.555,50.24],[3.51,50.34],[3.505,50.335],[3.47,50.34],[3.445,50.35],[3.365,50.34],[3.22,50.39],[3.18,50.42],[3.145,50.47],[3.125,50.55],[3.105,50.585],[3.1,50.61],[3.08,50.625],[3.05,50.62],[3.01,50.58],[2.97,50.555],[2.94,50.545],[2.87,50.545],[2.745,50.58],[2.69,50.61],[2.645,50.665],[2.615,50.665],[2.56,50.68],[2.485,50.75],[2.455,50.8],[2.44,50.94],[2.43,50.955],[2.42,51.005],[2.275,51.17]]]}

View File

@@ -0,0 +1,51 @@
#!/bin/python3
# from https://github.com/osmbe/openstreetmap-carto-be/blob/ansible/roles/render/templates/make_render_list.py
import mercantile
import requests
from shapely.geometry import (shape, box)
url = "http://polygons.openstreetmap.fr/get_geojson.py?id=52411&params=0.150000-0.005000-0.005000"
zooms = range(0, 18+1) # we make +1 to include zoom 18
# get the geojson
r = requests.get(url)
# get the shape
geom = shape(r.json())
bounds = geom.bounds
north = bounds[3]
south = bounds[1]
west = bounds[0]
east = bounds[2]
# print(f"north: { north }, south: { south }, east: { east }, west: { west }")
# print(zooms)
# cache the processed
processed = set()
# loop over all tiles
x = 0
for t in mercantile.tiles(west, south, east, north, zooms):
tile_bbox = mercantile.bounds(t)
bbox = box(tile_bbox.west, tile_bbox.south, tile_bbox.east, tile_bbox.north)
if geom.intersects(bbox):
x_ul = t.x - t.x % 8
y_ul = t.y - t.y % 8
if (x_ul, y_ul, t.z) not in processed:
print(f"{x_ul} {y_ul} {t.z}")
processed.add((x_ul, y_ul, t.z))
# for l in f.readlines():
# x_str, y_str, z_str = l.split(' ')
# # convert to in
# x_int = int(x_str)
# y_int = int(y_str)
# z_int = int(z_str)
# x_ul = x_int - x_int % 8
# y_ul = y_int - y_int % 8
#
# if (x_ul, y_ul, z_int) not in processed:
# print(f"{x_ul} {y_ul} {z_int}")
# processed.add((x_ul, y_ul, z_int))

View File

@@ -23,8 +23,50 @@ max_y = 50.77
min_x = 4.13
min_y = 49.49
# BE coordinates
min_z = 9
min_x = 2.54539
max_y = 51.50543
min_y = 49.49
max_x = 6.40
# BE buffer coordinates
min_x = 2.38246
max_y = 51.64907
min_y = 49.35438
max_x = 6.55565
#for low zoom levels BE
min_x = 0
max_y = 70
min_y = 20
max_x = 30
min_z = 2
max_z = 9
for z in range(min_z, max_z+1):
x1_tile, y1_tile = deg2num(max_y, max_x, z)
x2_tile, y2_tile = deg2num(min_y, min_x, z)
min_x_tile = x1_tile if x1_tile < x2_tile else x2_tile
max_x_tile = x1_tile if x1_tile > x2_tile else x2_tile
min_y_tile = y1_tile if y1_tile < y2_tile else y2_tile
max_y_tile = y1_tile if y1_tile > y2_tile else y2_tile
for tx in range(min_x_tile, max_x_tile):
if tx % 8 != 0:
continue
for ty in range(min_y_tile, max_y_tile):
if ty % 8 != 0:
continue
print("{} {} {}".format(tx, ty, z))
# BE buffer coordinates
min_x = 2.38246
max_y = 51.64907
min_y = 49.35438
max_x = 6.55565
min_z = 10
max_z = 18
for z in range(min_z, max_z+1):

105806
utils/tiles-BE-buffer.list Normal file

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff