Database persistence project group recommendation

James.White2@...
 

All,

 

In this morning’s Database Persistence project meeting, we started to review a comparison matrix of potential database options and started to explore performance evaluation testing of databases (either with our own or with a known and regarded benchmarck).

While the work (which could be significant) to explore database alternatives and performance test the alternatives against one another could continue, it seemed that at this time (targeting Edinburgh release), it might be more appropriate to focus on the service architecture as it relates to persistence and allow for easier swap of the database by 3rd parties while also providing at least one other alternative to start to give examples of how to implement EdgeX services with alternate persistence mechanism.  Therefore, I proposed the following plan, with no objection from the project group, to take to the core working group for consideration at the next meeting:

 

oFor Edinburgh release, the core dev team works to implement a layer/abstraction system for databases.  Allowing any database to be used under the covers more easily.  Getting rid of any MongoDB specific ID or BSON references and isolating MongoDB specific to replaceable segments of the code base.

oFor Edinburgh release, we implement 2 options for the EdgeX reference implementation:  a MongoDB and a Redis set of core services.  this based on the fact that Redis aligns with many of the requirements (and is not an embedded database), shows good performance in benchmark tests and has Redis members participating as part of the project currently.

oFor Edinburgh release, we well document how to replace the database for 3rd party replacements.

oFor Edinburgh release, we have a certification process for core services – insuring any new core service adheres to the API sets and work appropriately in an EdgeX environment

oFor Edinburgh release, we have a marketplace for core services (something already underway via many areas of the project)

oFor Edinburg release, we have a performance test harness and set of benchmarks for core services that provide statistics on memory usage, CPU usage, speed of queries, writes, etc. to give the community a sense indications that may help in choosing an implementation that aligns best with their use case.

oFor the Fuji release, we make a determination of which of the reference implementations we want to keep long term (may be one or both or even take one out of the marketplace if it is shows itself to be better)

 

If you have opinions or suggestions about this plan, please reply to this thread and/or participate in the upcoming core working group meeting.

 

Thanks

 

Jim White

Distinguished Engineer, IoT Platform Development Team Lead

EdgeX Foundry Technical Steering Committee Vice Chairman

Dell Technologies | IoT Solutions Division

Office +1 512-723-6139, mobile/text +1 612-916-6693

james_white2@...

 

Join EdgeX-DB-Persist@lists.edgexfoundry.org to automatically receive all group messages.