Fw: [Edgex-devel] Coupling Between Microservices
Re-posted to the other technical community mail lists for exposure to those not subscribed to the devel list.
From: edgex-devel-bounces@... <edgex-devel-bounces@...> on behalf of White2, James
Sent: Sunday, October 22, 2017 9:55 PM
Subject: Re: [Edgex-devel] [Edgex-golang] Coupling Between Microservices
It has been an exciting few weeks in EdgeX-land. With the IoT SWC show, the IIC announcement, announcement of the additional companies joining EdgeX, and our first community release Friday. I am also encouraged that over the past few weeks, there has been much interest and discussion from the community about how we make EdgeX better. This tells me, despite all the back and forth of any technical argument that we are collectively on the right track. This is how good products get made.
Before us now are some very interesting and important discussions about some of the fundamental elements of the EdgeX design and architecture. I think we have all accepted the most fundamental architectural design pattern for our platform: that of micro services. But of those micro services, how will they be constructed, managed, updated, etc.. Important questions!
As the original architect of Fuse, I have tried to be very careful to listen first and not trample on any design or architectural idea as it has so far emerged from the community (or even some that came from outside the community). Fuse was a proof-of-concept project and as such, no one more than I am aware that in some cases short cuts were taken in order to prove an idea and not necessarily build a production ready system. My deepest apologies if any comment I have made, or am about to make, would ever suggest I am not open to your suggestions to improve EdgeX.
Having said that, some of the considerations brought forth now (and in the past) were actually considered – and even tried – by the original Fuse team. So, I feel remiss in not at least providing that background on occasion.
Also, before starting the EdgeX Foundry open source effort, my former boss and I consulted with several successful open source project-leads to find out what helped make them successful. One overwhelming factor that all leads point to (perhaps the only consistent advice we got) was this: open source efforts are driven by actual submitted code. We were warned that people will want to debate designs, architectures, process, methodology, etc. in the technical working groups and committee meetings. Worse - we were warned of outsiders demanding change without contribution. As we were advised then, the court of opinion that drives open source development is corroborated by code. You are limited by what volunteers do for you. As they advised: if an idea is important enough to put into code, then it gets full attention and consideration, but until that code is delivered – it may be considered but only accepted when there is near unanimous opinion for a change or addition.
Given that background, I think we face 3 particular questions right now:
1) Should EdgeX micro services have shared libraries (at least shared libraries per language implementation).
2) Should EdgeX micro services have a single repo or multiple repos
3) What messaging infrastructure should EdgeX use between services when not using REST. Tactically, this question is being raised because of difficulties of using ZMQ between core data and export distro in Go, but strategically, I think this question will broaden as we go forward.
I’d like to weigh in on the first two questions in this mailer. I’d also like to suggest (per our last core working group meeting) that we start to tackle and finalize these in the working group meetings starting with the up coming meeting Thursday. Let’s continue to use the mailers to get opinions and ideas out, but then use the meetings to wrap up the discussions and make decisions when we can.
I have some opinions on the 3rd issue as well, but believe we all need a little more time to do some additional research and exchange of ideas as has been done on the first two issues.
I. Shared Libraries
On the issue of shared libraries…
I think Tony did an outstanding job of outlining the black, white and shades of gray that underline this design issue in light of micro services. Each of you have generally made eloquent arguments that fall in one or more of the categories. Software architecture rarely provides a clear and concise option with cut/dry decision points.
In Fuse, I actually started building the micro services with no shared code. In early micro services, all of the shared concepts implemented as Java classes were rewritten (or copied and pasted) into each micro service. I can tell you from my personal experience, this was a maintenance nightmare. Especially as the system was evolving (which I think EdgeX still is), it meant every time I had to change something like the concept of Device, I had to revisit each service to make the duplicative change. And it was not uncommon that something would be forgotten and a micro service got out of sync. Plus, as new supporting services, device service, etc. kept on increasing, I kept asking myself why was I replicating all this code – and those on my team wondered the same as they started to join me in development. Bloat was evident. Maintenance/refactoring sucked. Therefore, the first major refactor of Fuse included the development of some baseline libraries we see in EdgeX today: domain, exception, client libraries, etc. I became an ardent proponent of the DRY principal.
Having said that, when asked to replace the underlying database during a Thanksgiving break about a year ago, I was also very happy that we had very good separation of concerns – especially as it relates to various layers/technologies of the micro services. Replacing Mongo for a SQLite instance took me less than three days and all that work was just wresting with SQL code for the new DAO classes in core data. No code change in any other service was required. So, where we do not have those separations in the libraries today, that’s a problem than needs immediate attention.
II. Shared Libraries proposal per WG chair
Let me propose a somewhat happy (?) middle ground for us to work from and approach a solution to this first dilemma…
There is a lot of code already in EdgeX and I am not sure the argument to tear out libraries is painted with enough convincing evidence today to warrant a lot of micro service rework in that existing code – I don’t see a lot of value in that. In the core and supporting services of EdgeX – as the chair of the working group in charge of these micro services – I believe we’ll stick to the use of libraries and make those libraries available to others that want to use them (in Java, Go and other languages that we may be working on … yes this eludes to some secret work the Dell super genius has been doing 😊 ).
Further, as we get ready to provide security and system management functionality in the California release, I firmly believe we are going to have some features which will need to be built into each and every micro service and this is going to be more easily accomplished and tested by building shared libraries for this purpose (for example a base service that implements the required system management bootstrapping for each service).
However, outside of the core/supporting layers I believe we can and should use these areas for some architectural/design playgrounds and experimentation. I would submit that the working group chairs of the applications and device service areas, in particular, should be free to require libraries or allow the use of no libraries as they see fit. However, this liberty also comes with a requirement. Your efforts must keep up with the pace of EdgeX development. If you choose not to use shared libraries and a change is required to something like the Device concept (which is currently in a library) or we need to add security/system management features (again probably covered in a library) – I don’t think it fair to require people working on those changes or features to have to address additional coding in each of your non-library microservices. You have to take on that challenge and do so in a way that does not stall EdgeX from being delivered.
For those that accept the library approach – we (including myself here) have to live up to the responsibility to keep concerns separated. Case in point, the current Go libraries that tie to the Mongo database must be refactored to avoid such intrusion. When a team identifies a tight coupling, we must remove or reduce it.
III. Multi or single repos
Again, when I first started Fuse, we had just one repos. As more people entered the project, as more shared libraries started to exist (yes - the two issues are related) and as we saw the need for independent releases of micro services, it was clear that multiple code repositories were needed. The complexities of getting all the code and of having more complex build/CI processes was outweighed by the need to divide and conquer the work. It allowed for specialization without requiring someone to get the whole code base to work on a smaller service. It allowed isolation of small tests and trial of new technology to enter into one service without the requirement to enter the project as a whole. It also allowed for bringing in different versions of 3rd party libraries or frameworks as needed to address various needs in each micro service without concern to the others. And it allowed for true polyglot development.
Fuse was mostly Java. I realize and see firsthand that something like Go Lang uses a slightly different approach when developing an application. So here again, I recommend a similar suggestion to the last.
IV. Multi v single repos approach per WG
In the core and supporting services, there would be much rework for what I see as little gain to move to a single repo – for either the Java, Go or other languages right now. So, we’ll continue to use multiple repositories. But again, let’s let the WG chairs, especially in new services and or in new languages feel free to experiment with the alternate approach. This may be especially warranted as you develop replacement services in alternate languages (for example, put all the Go export services in a single repo).
I would ask that working group chairs work hard to keep your areas of EdgeX consist. If you think a single repos is the best approach for all of export, for example, then you should work to put all the existing Java export micro services under one repo as you also work the new Go or C services into one repo for those languages. Consistency will help keep the documentation and education of the community clean and the development more straight forward in those areas.
V. archive and replace options
In addition to my proposals for these two specific issues above, I also leave open the option - under the code is king philosophy - for someone with high ambition to provide EdgeX with a total rewrite of all the applicable services to demonstrate with code that one approach is better than the other. We’ll call this the nuclear-archive-and-replace option (cut from one release of EdgeX to another release of EdgeX that takes a fundamental shift in approach). This does not mean that if you build it we (the EdgeX community) have to accept it, but we have responsibility to at least look at the full replacement option you have provided. If you feel that strongly that your approach – in this case complete library-less EdgeX or whatever - is best and you demonstrate it with code that allows the community to swing in that direction immediately with little or no pain because they see the value in that approach, then we must be open to it, study it and vote it up or down.
We MUST always allow for the project to accept such a proposal, lest a group someday decide to fork all of EdgeX and go off and do their own thing with those believing in one way or the other (see Hudson versus Jenkins as a prime example).
VI. General themes of re-design/re-architecture
As you can hopefully tell from my long (sorry) opinion on these two issues, I am suggesting some underlying themes that I think should guide us going forward with issues like these:
1. We need to be open to new ideas – tried/proven at the peripheral and then adopted at the core when proven and the value returned by the idea is clear.
2. The working group chairs (technical project leadership) need to share in the responsibility and decision leadership of our EdgeX effort – especially those chairs where room for more experimentation and options exist. If you see a better way, feel free to take some chances! However, don’t leave the rest of us waiting for your area to deliver. Always experimenting and never delivering isn't going to work either.
3. The prime directive: decisions we make must be ruled by code. Existing code that works is far better than theoretically superior designed code that doesn’t yet exist. So, before we change EdgeX, experiment at the edges (no pun intended) and prove your results to the community in code and have a plan for how we work it into existing code where warranted (remember – there is a lot of existing code and functionality – so what’s your plan for implementing a significant change, especially when it may all fall on you to do it?).
From: edgex-devel-bounces@... <edgex-devel-bounces@...> on behalf of Drasko DRASKOVIC <drasko@...>
Sent: Friday, October 20, 2017 7:44 AM
To: Car, Boran
Cc: edgex-devel@...; edgex-golang@...
Subject: Re: [Edgex-devel] [Edgex-golang] Coupling Between Microservices
On Fri, Oct 20, 2017 at 1:35 PM, Car, Boran
I think it's time for a Jim moment: "So what I'm hearing is..." etc... to bring some conclusions :).Jim is wisely waiting that we shoot all our bullets before ;).
Possibly around the line that it's not all black and white, but I'm not going to even try it - I'd probably be a bad replacement.io
n and then generation.Thanks Boran - both your and Tony's inputs are very useful!
I have been thinking over this problematic, and I am inclined to say
that much of the difficulty comes from the coupling of domains
themselves, not necessarily from the implementation. Actually, EdgeX
code somewhat resembles me as a monolith broken into different repos,
rather then independent set of microservices. And I think that much of
this impressions comes from the nature of these services - these are
not some distinct architectural blocks - like for example pub/sub
broker or database would be, or Redis kv store. When you have these
kind of services, independent and self-sufficient, with different
responsibilities and task in the system and different nature then it
is easy to see clear separations - each of them is run in a separate
container, etc, etc...
However, whole EdgeX system can logically run very well as a monolith
app, and even do we introduce some separation that seems visible -
like between layer that does the export services and the core layer -
still there is reuse of the logic and data - both of these layers work
on the same task practically and I think domain coupling comes from
Having in mind this observation, I am not against sharing data models
between these microservices (some minor stuff can be duplicated, but
bigger dependencies would probably fit well in shared libs).
However, this also brings me back to one of my previous topics - monorepo :).
As I mentioned, having shared libs probably is not so bad in this
case, but having data models scattered over different repos can be
confusing. Let's take a concrete eample: you have microservice A that
uses some class, and also service B that use the same class. Where
would you define this class - in repo A or repo B? Or will you create
new repo C to be a meta-repo and hold data models only, these data
models that are used in both A and B.
And this is exactly the approach that Export Services take - both
Export Client (https://github.com/edgexfoundry/export-client) and
Export Distro (https://github.com/edgexfoundry/export-distro) use
models that are because of this fact that both services share them put
in a separate repo called Export Domain
(https://github.com/edgexfoundry/export-domain). And this looks OK so
far - but the problem is that these Java classes in Export Domain
include some references and dependencies on classes defined in EdgeX
Core (https://github.com/edgexfoundry/core-domain), and this is where
the things get a bit ugly, because while so far we were staying in the
same Export layer we now start sharing cross-domain data. But as I
mentioned, I am starting to think that this is actually not even
cross-domain, as these divisions between domains (layers) seem to be
The approach I took to lower the impact of this problem is that I've
put both EdgeX Client and EdgeX distro in the same repo:
https://github.com/drasko/edgex-export. Naturally - EdgeX Domain is
there also. But the good part is that nothing is lost but this move to
a single repo - there is still strong division between code - Client
stuff live in the `client` dir
(https://github.com/drasko/edgex-export/tree/master/client) and distro
stuff in `distro` dir
files hold the shared structures that are used in both client and
Note that we still have two distinct microservices, completely
independent of each other, built independently and deployed
independently via different Dockerfiles. Elegance of Go just helps us
organize the code better.
Note also - and this is very important - that this code separation
allowed two companies - Mainflux and Cavium - to workd completely
independently on differerent services. Mainflux contributed client
part, while Cavium is working on distro. Having a simple repo actually
help us sync on the common files using PRs and Issues of this GitHub
repo for discussion - for example:
In my opinion this speeds up development process and assures higher
quality and integrity of code.
I proposed earlier that we evaluate actually putting all EdgeX Go code
in one single repo, using this model that I proposed in export
servcies and following the practices from Google
(https://www.youtube.com/watch?v=W71BTkUbdqE) or Uber
But if this is to much integration, we can at least group the layers
that we have today - have separate repos for export, core, supporing
services and device services - just 4 repos for whole EdgeX Go code. I
think that will diminish the impact of shared cross-domain data
models, as all services from the same repo can reuse data defined in
the top-level files of the repo.
My $0.2 and sorry for the longer mail again, could not make it shorter.
Mainflux Author and Technical Advisor
www.mainflux.com | Industrial IoT Cloud
Engineering Division | Paris, France
EdgeX-Devel mailing list
EdgeX-Devel mailing list