Goals

Now that we have partitions, we can parallelise the creation of indexes so it takes a lot less time to create them instead of doing so sequentially.

Challenges

The indexes are a big matrix of 1s and 0s, being the columns the database row column and the rows the unique value of the rows for the particular table field.

We also keep a mapping from value to the row id in the bitmap. So, for that you need to keep track of the row ids.

The problem comes from the columns. The column id is sequentially incremented.

Ideas

We could share a global counter protected by a mutex. That way, there would be no need to pass the colID and keep it sequential. This has one downside, though: index creation is not idempotent and the order in which the rows are stored (and thus, returned) will be different every time you create the same index.

Also, we would need to stop saving batches and fields in the driver structure, as multiple operations might be taking place at the same time.

UPDATE: this can't be done or indexes can't be combined.