TL;DR

Is it possible that the data fails to be written but no exception is raised?

Hi.

I'm currently investigating a recurring data loss issue.

I've been searching on the bug tracker but couldn't find anything relevant. Like in #80 (comment), I get no error. Opening a new issue as this is not related to #80.

I'm using default parameters (except a batch size of 5000 but I don't think it matters).

My program reads timeseries from CSV files in a directory and sends the data to InfluxDB. Once done, it records the last file modification time to know where to start next time, so as to only process new files.

There is not much exception handling. If an exception is not caught, the program just crashes. If this happens, the last modification times are not updated and the program reloads all the files next time.

Apparently, the program doesn't crash, but there is missing data in the database. Batches of contiguous missing data. The size of the holes may vary and generally spans over several files.

Sometimes, files are half loaded, so it's not the time-based file filtering mechanism that skips a file it shouldn't.

If I re-run the program manually on incriminated files, the data is correctly loaded, so the files are correctly formed.

Hence, my question:

Is it possible that the data fails to be written but no exception is raised?

I understand there is a retry mechanism. I assume if influxdb_client raises no exception, then all data was correctly acknowledged by InfluxDB. There might have been retries but those were successful in the end.

Is this a safe assumption?

I do call __del__ on write_api and client, so I don't think it is a matter or flushing buffers. I tried once with debug=True and I could check there was no data read after the last POST to InfluxDB, so all data is sent before closing the program.

I wouldn't mind providing the code but it's a bit convoluted, not really a minimal example. Since we do this kind of things for different projects, I wrote a file importer class to handle the common part (database connection and file selection) and child classes just need to specify how to read the files and add ad-hoc tags from project-specific metadata. Not the ideal scenario to provide a reproducible case.

As such, this is not really a bug report, more of a clarification request. Please feel free to redirect me if this is not the right place to ask.

Thanks.