Compare commits

...

6 Commits
v1.2 ... v1.3.2

Author SHA1 Message Date
Alexander Overvoorde
80586aff5f Fix Postgres not being reachable from outside container 2019-09-22 16:39:56 +02:00
Alexander Overvoorde
b82e37b9b5 Improve compatibility of Apache logging locations 2019-09-17 22:03:03 +02:00
Alexander Overvoorde
451f335bc5 Add ALLOW_CORS parameter 2019-08-31 13:44:23 +02:00
Alexander Overvoorde
8744ab2792 Fix flat nodes explanation 2019-08-22 22:05:45 +02:00
Alexander Overvoorde
31bf2d718b Add a way to enable options like --flat-nodes (fixes #53) 2019-08-20 19:40:24 +02:00
Alexander Overvoorde
fcc6168253 Fix updating by moving to latest version of regional extract tool 2019-08-08 21:39:20 +02:00
5 changed files with 102 additions and 9 deletions

View File

@@ -131,17 +131,20 @@ RUN mkdir /var/lib/mod_tile \
&& mkdir /var/run/renderd \ && mkdir /var/run/renderd \
&& chown renderer /var/run/renderd && chown renderer /var/run/renderd
RUN echo "LoadModule tile_module /usr/lib/apache2/modules/mod_tile.so" >> /etc/apache2/conf-available/mod_tile.conf \ RUN echo "LoadModule tile_module /usr/lib/apache2/modules/mod_tile.so" >> /etc/apache2/conf-available/mod_tile.conf \
&& a2enconf mod_tile && echo "LoadModule headers_module /usr/lib/apache2/modules/mod_headers.so" >> /etc/apache2/conf-available/mod_headers.conf \
&& a2enconf mod_tile && a2enconf mod_headers
COPY apache.conf /etc/apache2/sites-available/000-default.conf COPY apache.conf /etc/apache2/sites-available/000-default.conf
COPY leaflet-demo.html /var/www/html/index.html COPY leaflet-demo.html /var/www/html/index.html
RUN ln -sf /proc/1/fd/1 /var/log/apache2/access.log \ RUN ln -sf /dev/stdout /var/log/apache2/access.log \
&& ln -sf /proc/1/fd/2 /var/log/apache2/error.log && ln -sf /dev/stderr /var/log/apache2/error.log
# Configure PosgtreSQL # Configure PosgtreSQL
COPY postgresql.custom.conf.tmpl /etc/postgresql/10/main/ COPY postgresql.custom.conf.tmpl /etc/postgresql/10/main/
RUN chown -R postgres:postgres /var/lib/postgresql \ RUN chown -R postgres:postgres /var/lib/postgresql \
&& chown postgres:postgres /etc/postgresql/10/main/postgresql.custom.conf.tmpl \ && chown postgres:postgres /etc/postgresql/10/main/postgresql.custom.conf.tmpl \
&& echo "\ninclude 'postgresql.custom.conf'" >> /etc/postgresql/10/main/postgresql.conf && echo "\ninclude 'postgresql.custom.conf'" >> /etc/postgresql/10/main/postgresql.conf
RUN echo "host all all 0.0.0.0/0 md5" >> /etc/postgresql/10/main/pg_hba.conf \
&& echo "host all all ::/0 md5" >> /etc/postgresql/10/main/pg_hba.conf
# copy update scripts # copy update scripts
COPY openstreetmap-tiles-update-expire /usr/bin/ COPY openstreetmap-tiles-update-expire /usr/bin/
@@ -155,6 +158,8 @@ RUN chmod +x /usr/bin/openstreetmap-tiles-update-expire \
USER renderer USER renderer
RUN cd ~/src \ RUN cd ~/src \
&& git clone https://github.com/zverik/regional \ && git clone https://github.com/zverik/regional \
&& cd regional \
&& git checkout 612fe3e040d8bb70d2ab3b133f3b2cfc6c940520 \
&& chmod u+x ~/src/regional/trim_osc.py && chmod u+x ~/src/regional/trim_osc.py
# Start running # Start running

View File

@@ -79,6 +79,50 @@ docker run \
This will enable a background process that automatically downloads changes from the OpenStreetMap server, filters them for the relevant region polygon you specified, updates the database and finally marks the affected tiles for rerendering. This will enable a background process that automatically downloads changes from the OpenStreetMap server, filters them for the relevant region polygon you specified, updates the database and finally marks the affected tiles for rerendering.
### Cross-origin resource sharing
To enable the `Access-Control-Allow-Origin` header to be able to retrieve tiles from other domains, simply set the `ALLOW_CORS` variable to `1`:
```
docker run \
-p 80:80 \
-v openstreetmap-data:/var/lib/postgresql/10/main \
-e ALLOW_CORS=1 \
-d overv/openstreetmap-tile-server \
run
```
### Connecting to Postgres
To connect to the PostgreSQL database inside the container, make sure to expose port 5432:
```
docker run \
-p 80:80 \
-p 5432:5432 \
-v openstreetmap-data:/var/lib/postgresql/10/main \
-d overv/openstreetmap-tile-server \
run
```
Use the user `renderer` and the database `gis` to connect.
```
psql -h localhost -U renderer gis
```
The default password is `renderer`, but it can be changed using the `PGPASSWORD` environment variable:
```
docker run \
-p 80:80 \
-p 5432:5432 \
-e PGPASSWORD=secret \
-v openstreetmap-data:/var/lib/postgresql/10/main \
-d overv/openstreetmap-tile-server \
run
```
## Performance tuning and tweaking ## Performance tuning and tweaking
Details for update procedure and invoked scripts can be found here [link](https://ircama.github.io/osm-carto-tutorials/updating-data/). Details for update procedure and invoked scripts can be found here [link](https://ircama.github.io/osm-carto-tutorials/updating-data/).
@@ -94,6 +138,7 @@ docker run \
-d overv/openstreetmap-tile-server \ -d overv/openstreetmap-tile-server \
run run
``` ```
### AUTOVACUUM ### AUTOVACUUM
The database use the autovacuum feature by default. This behavior can be changed with `AUTOVACUUM` environment variable. For example: The database use the autovacuum feature by default. This behavior can be changed with `AUTOVACUUM` environment variable. For example:
@@ -105,6 +150,29 @@ docker run \
-d overv/openstreetmap-tile-server \ -d overv/openstreetmap-tile-server \
run run
``` ```
### Flat nodes
If you are planning to import the entire planet or you are running into memory errors then you may want to enable the `--flat-nodes` option for osm2pgsql. This option takes a path to a file that must be persisted so we should first set up a volume with the right permissions:
```
docker run -it -v openstreetmap-nodes:/nodes --entrypoint=bash overv/openstreetmap-tile-server
$ chown renderer:renderer -R /nodes
$ exit
```
You can then use it during the import process as follows:
```
docker run \
-v /absolute/path/to/luxembourg.osm.pbf:/data.osm.pbf \
-v openstreetmap-nodes:/nodes \
-v openstreetmap-data:/var/lib/postgresql/10/main \
-e "OSM2PGSQL_EXTRA_ARGS=--flat-nodes /nodes/flat_nodes.bin" \
overv/openstreetmap-tile-server \
import
```
### Benchmarks ### Benchmarks
You can find an example of the import performance to expect with this image on the [OpenStreetMap wiki](https://wiki.openstreetmap.org/wiki/Osm2pgsql/benchmarks#debian_9_.2F_openstreetmap-tile-server). You can find an example of the import performance to expect with this image on the [OpenStreetMap wiki](https://wiki.openstreetmap.org/wiki/Osm2pgsql/benchmarks#debian_9_.2F_openstreetmap-tile-server).
@@ -129,6 +197,10 @@ docker run \
``` ```
For too high values you may notice excessive CPU load and memory usage. It might be that you will have to experimentally find the best values for yourself. For too high values you may notice excessive CPU load and memory usage. It might be that you will have to experimentally find the best values for yourself.
### The import process unexpectedly exits
You may be running into problems with memory usage during the import. Have a look at the "Flat nodes" section in this README.
## License ## License
``` ```

View File

@@ -10,4 +10,8 @@
ErrorLog ${APACHE_LOG_DIR}/error.log ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined CustomLog ${APACHE_LOG_DIR}/access.log combined
<IfDefine ALLOW_CORS>
Header set Access-Control-Allow-Origin "*"
</IfDefine>
</VirtualHost> </VirtualHost>

View File

@@ -21,3 +21,5 @@ random_page_cost = 1.1
track_activity_query_size = 16384 track_activity_query_size = 16384
autovacuum_vacuum_scale_factor = 0.05 autovacuum_vacuum_scale_factor = 0.05
autovacuum_analyze_scale_factor = 0.02 autovacuum_analyze_scale_factor = 0.02
listen_addresses = '*'

20
run.sh
View File

@@ -2,13 +2,16 @@
set -x set -x
function CreatePostgressqlConfig() function createPostgresConfig() {
{
cp /etc/postgresql/10/main/postgresql.custom.conf.tmpl /etc/postgresql/10/main/postgresql.custom.conf cp /etc/postgresql/10/main/postgresql.custom.conf.tmpl /etc/postgresql/10/main/postgresql.custom.conf
sudo -u postgres echo "autovacuum = $AUTOVACUUM" >> /etc/postgresql/10/main/postgresql.custom.conf sudo -u postgres echo "autovacuum = $AUTOVACUUM" >> /etc/postgresql/10/main/postgresql.custom.conf
cat /etc/postgresql/10/main/postgresql.custom.conf cat /etc/postgresql/10/main/postgresql.custom.conf
} }
function setPostgresPassword() {
sudo -u postgres psql -c "ALTER USER renderer PASSWORD '${PGPASSWORD:-renderer}'"
}
if [ "$#" -ne 1 ]; then if [ "$#" -ne 1 ]; then
echo "usage: <import|run>" echo "usage: <import|run>"
echo "commands:" echo "commands:"
@@ -22,7 +25,7 @@ fi
if [ "$1" = "import" ]; then if [ "$1" = "import" ]; then
# Initialize PostgreSQL # Initialize PostgreSQL
CreatePostgressqlConfig createPostgresConfig
service postgresql start service postgresql start
sudo -u postgres createuser renderer sudo -u postgres createuser renderer
sudo -u postgres createdb -E UTF8 -O renderer gis sudo -u postgres createdb -E UTF8 -O renderer gis
@@ -30,6 +33,7 @@ if [ "$1" = "import" ]; then
sudo -u postgres psql -d gis -c "CREATE EXTENSION hstore;" sudo -u postgres psql -d gis -c "CREATE EXTENSION hstore;"
sudo -u postgres psql -d gis -c "ALTER TABLE geometry_columns OWNER TO renderer;" sudo -u postgres psql -d gis -c "ALTER TABLE geometry_columns OWNER TO renderer;"
sudo -u postgres psql -d gis -c "ALTER TABLE spatial_ref_sys OWNER TO renderer;" sudo -u postgres psql -d gis -c "ALTER TABLE spatial_ref_sys OWNER TO renderer;"
setPostgresPassword
# Download Luxembourg as sample if no data is provided # Download Luxembourg as sample if no data is provided
if [ ! -f /data.osm.pbf ]; then if [ ! -f /data.osm.pbf ]; then
@@ -52,7 +56,7 @@ if [ "$1" = "import" ]; then
fi fi
# Import data # Import data
sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /home/renderer/src/openstreetmap-carto/openstreetmap-carto.lua -C 2048 --number-processes ${THREADS:-4} -S /home/renderer/src/openstreetmap-carto/openstreetmap-carto.style /data.osm.pbf sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /home/renderer/src/openstreetmap-carto/openstreetmap-carto.lua -C 2048 --number-processes ${THREADS:-4} ${OSM2PGSQL_EXTRA_ARGS} -S /home/renderer/src/openstreetmap-carto/openstreetmap-carto.style /data.osm.pbf
# Create indexes # Create indexes
sudo -u postgres psql -d gis -f indexes.sql sudo -u postgres psql -d gis -f indexes.sql
@@ -69,10 +73,16 @@ if [ "$1" = "run" ]; then
# Fix postgres data privileges # Fix postgres data privileges
chown postgres:postgres /var/lib/postgresql -R chown postgres:postgres /var/lib/postgresql -R
# Configure Apache CORS
if [ "$ALLOW_CORS" == "1" ]; then
echo "export APACHE_ARGUMENTS='-D ALLOW_CORS'" >> /etc/apache2/envvars
fi
# Initialize PostgreSQL and Apache # Initialize PostgreSQL and Apache
CreatePostgressqlConfig createPostgresConfig
service postgresql start service postgresql start
service apache2 restart service apache2 restart
setPostgresPassword
# Configure renderd threads # Configure renderd threads
sed -i -E "s/num_threads=[0-9]+/num_threads=${THREADS:-4}/g" /usr/local/etc/renderd.conf sed -i -E "s/num_threads=[0-9]+/num_threads=${THREADS:-4}/g" /usr/local/etc/renderd.conf