Re: [WebDNA] two ideas for running a cluster of WebDNA servers
This WebDNA talk-list message is from 2019
It keeps the original formatting.
numero = 114882
interpreted = N
texte = 2510 --Apple-Mail=_A472EC4B-E3C8-452C-B191-35EC4E7D2812 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Nice idea Chris, however 30 seconds to replicate is far too slow in an = ecommerce or high usage client data entry site. To be usable the master would have to replicate the write to the slaves = at the same instant as the master receives the write command. The read = from the slaves is OK but would mean another level of infrastructure to = balance the load across the slave servers, so in effect a load balancer. If you were to build a multi server site sitting behind a load balancer = then you would need to user the master/slave model with a second load = balancer for the slaves. The strain on the master could very well become = heavy if you did not have a method of delivering the writes to only one = slave db instance, my thought is that if you had 4 slaves, then the = master would have to precess the write to each of them. Your query log = concept is very achievable here to help reduce the load on the master. If you just had one front-end server and the db servers were simply a = method of backing up then you could possibly use the first method you = described. I think that there are two scenarios with two different methods = required. As Ken mentioned either of the two methods I describe add another level = of processing that would take away from the performance benefits of = WebDNA, plus the cost of running the master & slaves and ensuring that = they all stay in sync. If you are looking to build a solution for a particular project, then I = would not go past using a managed MySQL server from AWS - as yet I have = not tried the ones at DigitalOcean. The Aurora solution that AWS provide = is very good and becomes a scaleable system that can be triggered by a = load point. For me I would not try to reinvent the wheel as much as my = WebDNA head says that multi server WebDNA db would be fantastic. I know that the MySQL has been dropped from WebDNA, maybe the idea would = be to build a module for WebDNA that is not reliant on Oracle. Or maybe = the open source PostgresSQL is the way to go with the ability to process = another language (WebDNA) via another language such as C++, Python and = others. Maybe even NoSQL could be an idea to look at then at least you = have a number of solutions including Amazon DynamoDB, Google BigTable, = Apache Cassandra & MongoDB. If the uptake of WebDNA is a goal, then I = think that it is imperative to target db systems that businesses have = knowledge of and are comfortable to use due to their wide global = support. When it gets down to cost of the solution a WebDNA server plus a managed = MySQL solution wins hands down over the cost of having the WebDNA server = plus a number of other instances providing platforms for the slaves. = Unless of course you have a dedicated server with the ability to run = virtual machines, but then again you are relying on one machine and if = that fails all is lost. Kind regards Stuart Tremain Pharoah Lane Software AUSTRALIA webdna@plsoftware.com.au > On 5 Oct 2019, at 22:40, christophe.billiottet@webdna.us wrote: >=20 > 1.- one front-end server that will redirect the requests through POST = or GET to a cluster of back-end servers. When the request is for = writing, then only one of the back-end servers is solicited (always the = same), when is is for reading, it could be any of the back-end servers. >=20 > The "writing" server is the master, the others are slaves. Every 30 = sec, through rsync, the master is copied to a slave, and the WebDNA = slave reloads the databases in RAM. Meanwhile, the front-end server will = direct the reading requests to the other back-end servers, and this = sequentially. >=20 > Another idea would be the same front-end server with the same backend = servers, and instead of using rsync, the front end server would POST the = request to each one of the back-end servers: no more master and slave. >=20 > Another idea would be to keep a log database of all the writing = requests, so, by reading this log file, all the "slaves" would get the = same information. Databases could even be rebuilt in case of necessity. >=20 > I am sure there are other solutions. Any other idea? >=20 >=20 > - chris >=20 >=20 >=20 >=20 > --------------------------------------------------------- > This message is sent to you because you are subscribed to > the mailing list talk@webdna.us > To unsubscribe, E-mail to: talk-leave@webdna.us > archives: http://www.webdna.us/page.dna?numero=3D55 > Bug Reporting: support@webdna.us --Apple-Mail=_A472EC4B-E3C8-452C-B191-35EC4E7D2812 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
Nice = idea Chris, however 30 seconds to replicate is far too slow in an = ecommerce or high usage client data entry site.
To be usable the master would have to = replicate the write to the slaves at the same instant as the master = receives the write command. The read from the slaves is OK but would = mean another level of infrastructure to balance the load across the = slave servers, so in effect a load balancer.
If you were to build a multi server = site sitting behind a load balancer then you would need to user the = master/slave model with a second load balancer for the slaves. The = strain on the master could very well become heavy if you did not have a = method of delivering the writes to only one slave db instance, my = thought is that if you had 4 slaves, then the master would have to = precess the write to each of them. Your query log concept is very = achievable here to help reduce the load on the master.
If you just had one = front-end server and the db servers were simply a method of backing up = then you could possibly use the first method you described.
I think that there are = two scenarios with two different methods required.
As Ken mentioned either = of the two methods I describe add another level of processing that would = take away from the performance benefits of WebDNA, plus the cost of = running the master & slaves and ensuring that they all stay in = sync.
If you = are looking to build a solution for a particular project, then I would = not go past using a managed MySQL server from AWS - as yet I have not = tried the ones at DigitalOcean. The Aurora solution that AWS provide is = very good and becomes a scaleable system that can be triggered by a load = point. For me I would not try to reinvent the wheel as much as my WebDNA = head says that multi server WebDNA db would be fantastic.
I know that the MySQL = has been dropped from WebDNA, maybe the idea would be to build a module = for WebDNA that is not reliant on Oracle. Or maybe the open source = PostgresSQL is the way to go with the ability to process another = language (WebDNA) via another language such as C++, Python and others. = Maybe even NoSQL could be an idea to look at then at least you = have a number of solutions including Amazon DynamoDB, Google BigTable, = Apache Cassandra & MongoDB. If the uptake of WebDNA is a goal, then = I think that it is imperative to target db systems that businesses have = knowledge of and are comfortable to use due to their wide global = support.
When = it gets down to cost of the solution a WebDNA server plus a managed = MySQL solution wins hands down over the cost of having the WebDNA server = plus a number of other instances providing platforms for the slaves. = Unless of course you have a dedicated server with the ability to run = virtual machines, but then again you are relying on one machine and if = that fails all is lost.
Kind regards
Stuart Tremain
Pharoah Lane Software
AUSTRALIA
1.- = one front-end server that will redirect the requests through POST or GET = to a cluster of back-end servers. When the request is for writing, then = only one of the back-end servers is solicited (always the same), when is = is for reading, it could be any of the back-end servers.
The "writing" server is the master, the others are slaves. = Every 30 sec, through rsync, the master is copied to a slave, and the = WebDNA slave reloads the databases in RAM. Meanwhile, the front-end = server will direct the reading requests to the other back-end servers, = and this sequentially.
Another idea would = be the same front-end server with the same backend servers, and instead = of using rsync, the front end server would POST the request to each one = of the back-end servers: no more master and slave.
Another idea would be to keep a log database of all the = writing requests, so, by reading this log file, all the "slaves" would = get the same information. Databases could even be rebuilt in case of = necessity.
I am sure there are other = solutions. Any other idea?
- = chris
---------------------------------------------------------
This message is sent to you because you are subscribed to
the mailing list talk@webdna.us
To unsubscribe, E-mail to: = talk-leave@webdna.us
archives: http://www.webdna.us/page.dna?numero=3D55
Bug= Reporting: support@webdna.us
= --------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list talk@webdna.us To unsubscribe, E-mail to: talk-leave@webdna.us archives: http://www.webdna.us/page.dna?numero=3D55 Bug Reporting: support@webdna.us --Apple-Mail=_A472EC4B-E3C8-452C-B191-35EC4E7D2812-- .
Associated Messages, from the most recent to the oldest:
2510 --Apple-Mail=_A472EC4B-E3C8-452C-B191-35EC4E7D2812 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Nice idea Chris, however 30 seconds to replicate is far too slow in an = ecommerce or high usage client data entry site. To be usable the master would have to replicate the write to the slaves = at the same instant as the master receives the write command. The read = from the slaves is OK but would mean another level of infrastructure to = balance the load across the slave servers, so in effect a load balancer. If you were to build a multi server site sitting behind a load balancer = then you would need to user the master/slave model with a second load = balancer for the slaves. The strain on the master could very well become = heavy if you did not have a method of delivering the writes to only one = slave db instance, my thought is that if you had 4 slaves, then the = master would have to precess the write to each of them. Your query log = concept is very achievable here to help reduce the load on the master. If you just had one front-end server and the db servers were simply a = method of backing up then you could possibly use the first method you = described. I think that there are two scenarios with two different methods = required. As Ken mentioned either of the two methods I describe add another level = of processing that would take away from the performance benefits of = WebDNA, plus the cost of running the master & slaves and ensuring that = they all stay in sync. If you are looking to build a solution for a particular project, then I = would not go past using a managed MySQL server from AWS - as yet I have = not tried the ones at DigitalOcean. The Aurora solution that AWS provide = is very good and becomes a scaleable system that can be triggered by a = load point. For me I would not try to reinvent the wheel as much as my = WebDNA head says that multi server WebDNA db would be fantastic. I know that the MySQL has been dropped from WebDNA, maybe the idea would = be to build a module for WebDNA that is not reliant on Oracle. Or maybe = the open source PostgresSQL is the way to go with the ability to process = another language (WebDNA) via another language such as C++, Python and = others. Maybe even NoSQL could be an idea to look at then at least you = have a number of solutions including Amazon DynamoDB, Google BigTable, = Apache Cassandra & MongoDB. If the uptake of WebDNA is a goal, then I = think that it is imperative to target db systems that businesses have = knowledge of and are comfortable to use due to their wide global = support. When it gets down to cost of the solution a WebDNA server plus a managed = MySQL solution wins hands down over the cost of having the WebDNA server = plus a number of other instances providing platforms for the slaves. = Unless of course you have a dedicated server with the ability to run = virtual machines, but then again you are relying on one machine and if = that fails all is lost. Kind regards Stuart Tremain Pharoah Lane Software AUSTRALIA webdna@plsoftware.com.au > On 5 Oct 2019, at 22:40, christophe.billiottet@webdna.us wrote: >=20 > 1.- one front-end server that will redirect the requests through POST = or GET to a cluster of back-end servers. When the request is for = writing, then only one of the back-end servers is solicited (always the = same), when is is for reading, it could be any of the back-end servers. >=20 > The "writing" server is the master, the others are slaves. Every 30 = sec, through rsync, the master is copied to a slave, and the WebDNA = slave reloads the databases in RAM. Meanwhile, the front-end server will = direct the reading requests to the other back-end servers, and this = sequentially. >=20 > Another idea would be the same front-end server with the same backend = servers, and instead of using rsync, the front end server would POST the = request to each one of the back-end servers: no more master and slave. >=20 > Another idea would be to keep a log database of all the writing = requests, so, by reading this log file, all the "slaves" would get the = same information. Databases could even be rebuilt in case of necessity. >=20 > I am sure there are other solutions. Any other idea? >=20 >=20 > - chris >=20 >=20 >=20 >=20 > --------------------------------------------------------- > This message is sent to you because you are subscribed to > the mailing list talk@webdna.us > To unsubscribe, E-mail to: talk-leave@webdna.us > archives: http://www.webdna.us/page.dna?numero=3D55 > Bug Reporting: support@webdna.us --Apple-Mail=_A472EC4B-E3C8-452C-B191-35EC4E7D2812 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
Nice = idea Chris, however 30 seconds to replicate is far too slow in an = ecommerce or high usage client data entry site.
To be usable the master would have to = replicate the write to the slaves at the same instant as the master = receives the write command. The read from the slaves is OK but would = mean another level of infrastructure to balance the load across the = slave servers, so in effect a load balancer.
If you were to build a multi server = site sitting behind a load balancer then you would need to user the = master/slave model with a second load balancer for the slaves. The = strain on the master could very well become heavy if you did not have a = method of delivering the writes to only one slave db instance, my = thought is that if you had 4 slaves, then the master would have to = precess the write to each of them. Your query log concept is very = achievable here to help reduce the load on the master.
If you just had one = front-end server and the db servers were simply a method of backing up = then you could possibly use the first method you described.
I think that there are = two scenarios with two different methods required.
As Ken mentioned either = of the two methods I describe add another level of processing that would = take away from the performance benefits of WebDNA, plus the cost of = running the master & slaves and ensuring that they all stay in = sync.
If you = are looking to build a solution for a particular project, then I would = not go past using a managed MySQL server from AWS - as yet I have not = tried the ones at DigitalOcean. The Aurora solution that AWS provide is = very good and becomes a scaleable system that can be triggered by a load = point. For me I would not try to reinvent the wheel as much as my WebDNA = head says that multi server WebDNA db would be fantastic.
I know that the MySQL = has been dropped from WebDNA, maybe the idea would be to build a module = for WebDNA that is not reliant on Oracle. Or maybe the open source = PostgresSQL is the way to go with the ability to process another = language (WebDNA) via another language such as C++, Python and others. = Maybe even NoSQL could be an idea to look at then at least you = have a number of solutions including Amazon DynamoDB, Google BigTable, = Apache Cassandra & MongoDB. If the uptake of WebDNA is a goal, then = I think that it is imperative to target db systems that businesses have = knowledge of and are comfortable to use due to their wide global = support.
When = it gets down to cost of the solution a WebDNA server plus a managed = MySQL solution wins hands down over the cost of having the WebDNA server = plus a number of other instances providing platforms for the slaves. = Unless of course you have a dedicated server with the ability to run = virtual machines, but then again you are relying on one machine and if = that fails all is lost.
Kind regards
Stuart Tremain
Pharoah Lane Software
AUSTRALIA
1.- = one front-end server that will redirect the requests through POST or GET = to a cluster of back-end servers. When the request is for writing, then = only one of the back-end servers is solicited (always the same), when is = is for reading, it could be any of the back-end servers.
The "writing" server is the master, the others are slaves. = Every 30 sec, through rsync, the master is copied to a slave, and the = WebDNA slave reloads the databases in RAM. Meanwhile, the front-end = server will direct the reading requests to the other back-end servers, = and this sequentially.
Another idea would = be the same front-end server with the same backend servers, and instead = of using rsync, the front end server would POST the request to each one = of the back-end servers: no more master and slave.
Another idea would be to keep a log database of all the = writing requests, so, by reading this log file, all the "slaves" would = get the same information. Databases could even be rebuilt in case of = necessity.
I am sure there are other = solutions. Any other idea?
- = chris
---------------------------------------------------------
This message is sent to you because you are subscribed to
the mailing list talk@webdna.us
To unsubscribe, E-mail to: = talk-leave@webdna.us
archives: http://www.webdna.us/page.dna?numero=3D55
Bug= Reporting: support@webdna.us
= --------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list talk@webdna.us To unsubscribe, E-mail to: talk-leave@webdna.us archives: http://www.webdna.us/page.dna?numero=3D55 Bug Reporting: support@webdna.us --Apple-Mail=_A472EC4B-E3C8-452C-B191-35EC4E7D2812-- .
Stuart Tremain
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
SFTP (2004)
BUG ? Running on Empty (1999)
[Sum] function? (1997)
Authenticate and IIS (1997)
[WebDNA] Triggers.db wiped completely blank (2011)
Interfacing WebMerchant to www.fedex.com (1997)
[WebDNA] Grep everything and replace with asterisks? (2010)
[WebDNA] Deliminating encrypted values (2008)
[WriteFile] problems (1997)
Email notification to one of multiple vendors ? (1997)
question: back button prevention (1997)
File info on files with funky characters in their names (2003)
problems with 2 tags (1997)
shipcost (1997)
WebCat2 several catalogs? (1997)
WebDNA 6 (2004)
WebCat2b12--[searchstring] bug (1997)
Bug Report, maybe (1997)
SMSI (Scott) clues?? Bug fixes for next WebDNA release (2005)
MATH (1998)