Skip to content

mbtiler: use Wall mode and create index on tiles #29

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
DoFabien wants to merge 3 commits into
base: master
Choose a base branch
from
Open

mbtiler: use Wall mode and create index on tiles #29

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f217c56
mbtiler: use Wall mode and create index on tiles
DoFabien Feb 25, 2021
e8432ae
mbtiler : commit AFTER all insert
DoFabien Feb 25, 2021
6b51e36
commit data every 1000 tiles (about 150 Mo)
DoFabien Feb 25, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion rio_rgbify/mbtiler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -317,6 +317,8 @@ def run(self, processes=4):

# create a connection to the mbtiles file
conn = sqlite3.connect(self.outpath)
# Wall mode : Speedup by 10 the speed of writing in the database
conn.execute('pragma journal_mode=wal')
cur = conn.cursor()

# create the tiles table
Expand All @@ -325,6 +327,11 @@ def run(self, processes=4):
"(zoom_level integer, tile_column integer, "
"tile_row integer, tile_data blob);"
)

# create index on tiles for efficient access to this table
cur.execute(
"CREATE UNIQUE INDEX IF NOT EXISTS tile_index on tiles (zoom_level, tile_column, tile_row);")

# create empty metadata
cur.execute("CREATE TABLE metadata (name text, value text);")

Expand Down Expand Up @@ -366,6 +373,7 @@ def run(self, processes=4):
constrained_bbox = list(mercantile.bounds(self.bounding_tile))
tiles = _make_tiles(constrained_bbox, "epsg:4326", self.min_z, self.max_z)

tilesCount = 0
for tile, contents in self.pool.imap_unordered(self.run_function, tiles):
x, y, z = tile

Expand All @@ -379,8 +387,13 @@ def run(self, processes=4):
"VALUES (?, ?, ?, ?);",
(z, x, tiley, buffer(contents)),
)
tilesCount = tilesCount + 1
# commit data every 1000 tiles (about 150 Mo)
# Otherwise, the file .mbtiles-wal becomes huge (same size as the final file). The result is the need to have twice the size of the final Mbtiles on the hard drive
if (tilesCount % 1000 == 0):
conn.commit()

conn.commit()
conn.commit()

conn.close()

Expand Down