mercredi 6 juillet 2016

Sync MySQL inserts, updates, and delete operations with ElasticSearch

Here is a rather common case for elasticsearch(ES): as ES is near realtime, we want to sync it with our MySQL as much close to real time as possible. The initial tables are imported into index and it is working fine, but now we need to track inserting, updating and deleting new records and here is the place we have a question to. Import is done by elasticsearch-jdbc importer. It can have an interval parameter to periodically poll data, but: we cannot add fields to db which will be responsible for fetching new/updated rows, we want to have more atomic operations like DB insert row->ES insert document, DB update row->ES update document and so on. Two options which might meet our needs: Logstash with logstash-input-jdbc plugin Databus for MySQL from LinkedIn Other solutions are welcome!

Aucun commentaire:

Enregistrer un commentaire