We can define "pure cloud" approach as one in which IT services are provided:
- via public Internet or other WAN network
- by an external company
From the historical standpoint "pure cloud" providers represent the return to the mainframe era on a new technology level. Cloud providers are new "glass data centers" and due to overcentralization will soon will be hated as much if not more. The bottleneck is WAN links, and they are costly both in public and private clouds.
An interesting technology that now is developing as a part of "hybrid cloud" technologies is so called "edge computing". Of course, now the centralization train is still running at full speed, but in two to three years people might start asking questions "Why my application freezes so often".
There a synergy between "edge computing" and "grid computing": the latter can be viewed as grid computing at the edges of the network with the central management node (the headnode in �HPC speak�) software like Puppet, Chef or Ansible instead of HPC scheduler.
As WAN bandwidth provided at night is several times higher than during the day, one important task that can be performed by "edge computing" and the �time shift� activities: the delay of some transmissions and synchronization of data to the night time. UDP based tools for night allow to transfer large emopunt of data by saturating WAN bandwidth, but even TCP is can be usable at night on high latency links, if you can split the data stream into several sub streams. I reached comparable to UDB speeds when I was able to split the data stream into 8-16 sub-streams. Those "time shift" schemes does not require large investment. All �edge� resources should use unified images for VM and unified patching scheme.
I would define "hybrid cloud" as a mixture of private cloud, public cloud and "edge" computing with the emphasis on an optimizing consumed WAN bandwidth and offloading processes that consume WAN bandwidth at day time (One perverted example of WAN bandwidth usage is Microsoft Drive synchronization of user data) to the edge of the network via installation of "autonomous datacenters." Which can be just one remotely controlled. No local personnel is needed, so this is still "cloud-style organization" as all the staff is at central location. For the specific task of �shifting synchronization of user data� you need just one server with storage oer site.
From Wikipedia("edge computing"):
Edge computing is a distributed computing paradigm in which computation is largely or completely performed on distributed device nodes known as smart devices or edge devices as opposed to primarily taking place in a centralized cloud environment. The eponymous "edge" refers to the geographic distribution of computing nodes in the network as Internet of Things devices, which are at the "edge" of an enterprise, metropolitan or other network. The motivation is to provide server resources, data analysis and artificial intelligence ("ambient intelligence") closer to data collection sources and cyber-physical systems such as smart sensors and actuators.
Edge computing is seen as important in the realization of physical computing, smart cities, ubiquitous computing and the Internet of Things.
Edge computing about centrally (remotely) managed autonomous mini datacenters. Like in classic cloud environment staff is centralized and all computers are centrally managed. But they are located on sites not in central datacenter and that alleviate WAN bottleneck, for certain applications eliminating it completely. Some hardware producers already have product that are designed for such use cases.
... ... ...
Edge computing and grid computing are related.
IBM acquisition of Red Hat is partially designed to capture large enterprises interest in "private cloud" space, when virtual machines that constitute cloud are running on local premises in para-virtualization mode (Solaris x86 Zones, Docker containers). This way one has the flexibility of "classic" (off site) cloud without prohibitive costs overruns. On Azure a minimal server (2 core CPU, 32GB of space and 8GB of RAM) with minimal, experimental usage costs about $100 a month. You can lease $3000 Dell server with full hardware warranty for three years for $98.19 a month. With promotion you can even get free delivery and installation for this amount of money and free removal of hardware after three years of usage. So like with cloud you do not touch hardware, but you do not pay extra to Microsoft for this privilege either (you do have air-conditioning and electricity costs, though). And such "autonomous, remotely controlled, mini-datacenter" can be installed anywhere where Dell (or HP) provide services.
There are now several hardware offerings for edge computing. See for example:
Unfortunately, centralization drive now still rules. But having some "autonomous units" on the edge of network is an attractive idea that has future. First of all, because it allows to cut the required WAN bandwidth. Second, if implemented as small units with full remote management ("autonomous"), it allows to avoid many problems with "classic cloud" WAN bottleneck and problems typical for corporate datacenters.
Sooner or later the age of centralized cloud will come to the logical end. Something will replace it. As soon as the top management realizes how much they are paying for WAN bandwidth there will be a backlash. I hope that hybrid cloud might become a viable technical policy in two to three years� timeframe.
And older folks remember quite well how much IBM was hated in 60th and 70th (and this despite excellent compilers they provided innovative VM/360 OS which pioneered multitasking in the USA; base OS of VM/360 used REXX as shell) and how much energy and money people have spent trying to free themselves from this particular "central provider of services" model: "glass datacenter". It is not unfeasible that cloud provider will repeat a similar path on a new level. Their strong point: delivering centralized services for mobile users is also under scrutiny as NSA and other intelligence agencies have free access to all user emails.
Already now in many large enterprises WAN constitute a bottleneck, with Office 365 applications periodically frozen, and quality of service deteriorates to the level of user discontent, if not a revolt.
People younger then 40 probably can't understand to what extent rapid adoption of IBM PC was driven by the desire to send a particular "central service provider" to hell. That historical fact raises a legitimate question of the user resistance to "public cloud" model. Now security consideration and widespread NSA interception of traffic, geo-tracking of mobile phones and unlimited unrestricted access to major cloud providers webmail also became hotly debated issues.
WAN bandwidth is the bottleneck; Peter Deitch "eight classic fallacies"
The key selling point of edge computing for the enterprise datacenter is that the remote control technology now reached the level on which autonomous small datacenters -- datacenters with zero local support personnel are not only feasible but can compete with the classic cloud model, because this model has one grave deficiency -- WAN bandwidth as the bottleneck.
Like any new service along with solving old problems public cloud computing creates new, often more complex and less solvable. Proponents often exaggerate positive features and underestimate problems and costs. In no way cloud services as exemplified by Azure and Amazon are cheap. On Azure a minimal server (2 core CPU, 32GB of space and 8GB of RAM with minimal, experimental usage costs about $100 a month). You can lease $3000 Dell server for $98.19 a month. With promotion you can get free delivery and installation.
The vision of the IT future based on a remote centralized and outsourced datacenter that provides services via "cloud" using high speed fiber links is utopian as WAN can't scale. It is very true that the huge bandwidth of fiber optic lines made "in the cloud" computing more acceptable and some business models impossible in the past quite possible (Netflix). But that does not mean that this technology is a "universal bottle opener". Bottlenecks remain and enterprises now spend quite a bit of money fighting them. I would like to stress that this is just another example of centralized computing model (with usage of WAN connection instead of LAN connection for clients) and as such it has its limits.
According to Wikipedia "The Fallacies of Distributed Computing" there exists a set of common but flawed assumptions made by programmers in development of distributed, connected via network applications. They were originated by Peter Deitch and his "eight classic fallacies" can be summarized as following:
- The network is reliable.
- Latency is zero.
- Bandwidth is infinite.
- The network is secure.
- Topology doesn't change.
- There is one administrator.
- Transport cost is zero.
- The network is homogeneous.
Giants of cloud computing such as Amazon, Google, Yahoo and Microsoft push the idea of "pure" centralized "cloud-based" services to the extreme. They essentially are trying to replicate mainframe environment on a new level. Inheriting all the shortcoming of such an approach. Many companies is such industries as aerospace, financial services, chemical and pharmaceutical industries are trying to avoid "pure" cloud computing model because of their trade secrets, patented formulas and processes as well as compliance concerns. Also large companies usually run multiple datacenters and it is natural for them to be integrated into private "gated" cloud. No rocket science here.
These private clouds use a dedicated private network that connect their members and that help to avoid the problem of "bandwidth communism". At the same time a private cloud can consolidate the hardware and software needs of multiple smaller sites, provide load balancing and better economies of scale.
Costs of cloud servers are high. Using a small one socket "disposable" server and automatic provisioning and remote control tools (DRAC, ILO) you can achieve the same result in remote datacenters at a fraction of costs. This new idea of disposable server (a small 1U server now costs $1000 or less) is a novel idea that gives the second life to the idea of "autonomous datacenter" (at one point promoted by IBM, but soon completely forgotten as modern IBM is a fashion driven company ;-) and is a part of hybrid cloud model when some services like email are highly centralized, but other like file services are not.
The problem with "pure cloud" is the bandwidth cost money. That's why the idea of "backup to the cloud" (which in reality simply means backup over WAN) is such a questionable idea. Unless you can do it at night and did not splash to working hours you compete for bandwidth with other people and applications. It still can be used for private backups if you want to say good buy to your privacy.
But in an enterprise environment if such a backup spills over to morning hours you can make life of your staff really miserable, because they are depending of other applications which are also "in the cloud". This is classic Catch 22 situation with such a backup strategy. It is safer to do it "on site" and if due to regulations you need to have off site storage is probably cheaper to buy a private optical link (see, for example,
AT&T Optical Private Link Service) to a suitable nearby building. BTW a large attached storage box from, say, Dell costs around $40K for 280TB, $80K for 540TB and so on, while doubling the bandwidth of your WAN connection can run you a million a year.
For this amount of money you move such a unit to a remote site each week (after full backup is done) using a mid size SUV (the cost of SUV is included ;-). For more fancy setup you can use ideas from container based datacenters and use two cars instead of one (one on remote site and the other at main datacenter) for weekly full backup (Modular data center - Wikipedia ). Differential backup usually are not that large and can be handles via wire. If not, then USPS or FedEx is your friend. You will still money left of a a couple of lavish parties in comparison with your "in the cloud" costs.
Yes there are some crazy people who are trying WAN transferees of, say, 50-100TB of data thinking that this is a new way to do backups or synch two remote datacenters. They typically pay arm and leg for some overhyped and overpriced software from companies like IBM that uses UDP to speed transfer over lines with high latency. Such huge site-to-site transfer can't be saved with such a software no matter what presentation IBM and other vendors will give to your top brass (which usually does not understands WAN networking, or networking at all; the deficiently on which IBM relay for a long long time ;-). But multicast feature of such software is a real breakthrough and if you need to push the same file (for example a large videofile) to multiple sites for all employees to watch, it is really great).
But you can't change basic limitations of WAN. Yes some gains can be achieved by using UDP instead of TCP/IP, better compression, and using rsync-style protocol to avoid moving of identical blocks of data. But all those methods are perfectly applicable to local data transfers. And on WAN you typically are facing 3-5MB/sec links (a 10% fraction of 1Gbit link in best case). Now please calculate how much time it will take to transferee 50-100TB of data over such a link. On local 10Mbit link you can probably get 500 MB/sec. So the difference is speed is 100 times, give of take.
There is also a complimentary trend of creating remote autonomous datacenters controlled and managed from the well equipped central command&control center. Server remote control tool now reached the stage than you can mange server hundreds miles from your office as well as if you can walk to the server room. Actually not quite, and here much depends on your ingenuity.
But having a local servers with a central control center like is common of blade enclosures instead of centralized in some disctant location serve farm is more flexible approach as you can provide computer power directly were it is needed at the cost of a typical large server hosting provider.
Distributed scheme also requires much less WAN bandwidth and is more reliable as in case of problems with WAN connectivity. for example in case of hurricane that cut electricity supply in the central office, local services remain functional. IBM in the past promoted this approach under the name of "autonomic computing". Without much success. Probably due IBM level costs and marketing involved.
The level of automation strongly influences how efficiently IT is running. This factor alone might have greater influence on the future shape of IT than then all this "in the cloud" buzz. Vendors propose multiple solutions know as Enterprise System Management (ESM), computer grid (Sun), etc. The vision here are autonomous centrally controlled virtual instances controlled from a common control center. This approach can be combined with several other alternatives to "in the cloud" computing, such as "datacenter in the box", virtual appliances, and application streaming.
One important advantage of hybrid approach is the possibility of using autonomous servers. This is pretty new development that became possible due to advances in hardware (DRAC 7, blades) and software (Virtual Software Appliances). By autonomous server we will understand a server installed in location where there is no IT staff. There can be local staff knowledgeable on the level of a typical PC user, but that's it. Such servers are managed centrally from the central enterprise-wide location which retains highly qualified staff. Such location can be even on a different continent, if necessary.
This type of infrastructure is pretty common for large chemical companies which might have a hundred or more factories, storage depot, research labs, etc), banks (with multiple branches) and many other companies with distributed infrastructure. Details of implementation of course differ. Here we will concentrate on common features rather then differences. The key advantage of local autonomous servers (and by extension small local autonomous datacenters) is that local services and local traffic remains local. The latter provides several important advantages. Among them:
- You are less dependent on the reliability of central infrastructure of your "cloud provider" or company "master" datacenter hosting private cloud (which can well be across Atlantic). In case of "hybrid cloud" you can provide bandwidth-intensive services over local LAN via local real or virtual servers. Only administration is centralized.
- Snooping of traffic is more difficult as processing is distributed and, for example, if user mailboxes exist at local locations, not at central location where you can pick up all the information in one swipe.
- Cost of such infrastructure is lower then central cloud infrastructure
- Dependency on third parties is lower and you can control more aspects of it.
The attraction of cutting WAN traffic is directly connected with advances of modern hardware and software:
- Modern hardware provides reliable remote administration with virtual console and good remote management capabilities build-in (especially in blade environment); there are advances in virtual machine hypervisors, which allow running virtual instances at speed close to the speed of "native" OS, especially Solaris zones and linux containers. A typical Dell or HP server easily endure fives years of service without single minute of hardware-related downtime.
- Services from modern manufacturers have built-in KVM (DRAC in Dell, ILO in HP) with remote control of power and many bell and whistles. Dell DRAC is the standard in the industry as for capabilities (especially in blade environment) with DRAC 7 outperforming HP ILO by a wide margin.
- Raid configurations with additional redundant drive and/or two small/cheap (64GB or 128 GB) mirrored SSDs as a separate OS drive are essentially bullet-proof. Chances that they can cause downtime are very slim.
- Redundant power supplies usually last five-seven years without malfunctioning.
- Modern software now provide powerful tools for managing multiple similar servers from the central location dramatically lowering the cost of "reverse cloud" administration.
Problems with "pure" cloud computing
Cloud computing in it "pure" form (100% on remote datacenter) is centralized mainframe-style solution and it suffers from all limitations inherent in centralization. All powerful remote datacenters with minimum local client functionality are actually pretty costly and savings sited if advertisements are more often then not pure fiction. They are attractive for higher management as "stealth" outsourcing solution, but that's it. As any outsourced solution it suffers from loyalty problem. Cole combination of local and centralized capabilities are much more optimal. There are several important concerns with "pure" centralized solution (single remote datacenters-based cloud provider):
- Application performance concerns (WAN bandwidth issues; capability of remote datacenters to provide adequate response time, etc)
- Bureaucratization problems, sweeping under the carpet mainframe-style problems with external providers of IT services (everything becomes contract negotiation).
- Interoperability problems. Integration issues and balkanization threat in case of multiple "in the cloud" vendors. The lock-in concerns as well as potential rip-off due to complicated pricing models
- Total cost of ownership concerns. Cloud providers are pretty greedy and the long time cost can't be assumed to be stable, it can rise substantially as soon as the vendor feels that the customer locked into solution, evaporating all potential savings.
- Security concerns. Those are simply huge, from that danger of government interception of all data to competitors stealing the crown jewels of technology.
- Capabilities of application and application performance concern. Even the most powerful remote datacenter has difficulties to compete with model 3GHz 16GB of memory and 400GB SSD laptop.
- Poor customer service and increased risk of unexpected and damaging downtime. For cloud provider you are just one of many clients and in case of downtime you are typically left in the cold. As downtime is couse by problem in mind boggling complex giant infrastructure (typiclaly this is network problems) they are often slow to resolve depite of concentration of high level specialists.
Cloud computing as a technology is already more then ten years old ;-) and now we know quite a bit about its limitations.
While an interesting form of reincarnation of mainframes, it is critically dependent on availability of high speed fiber WAN networks. But this technology can be implemented in many flavors: can be a pure play (everything done on the servers of the cloud provider, for example Amazon or Google) or mixed approach with a significant part of the processing done at the client (more common and more practical approach). In no way it presuppose simplistic variant in which the everything should be done of the server while for some strange reasons rich client considered to be not kosher.
|In no way "in the cloud" software services model presuppose the everything should be done of the server and that rich powerfule client is not kosher. Also it ignores low cost of modern server and their excellent remote administration capabilities |
What is more important is providing Software as a Service (SaaS) . SaaS is a model of software deployment where an application is hosted as a service provided to customers across the network (local or Internet). By eliminating the need to install and run the application on the customer's own computer, SaaS alleviates the customer's burden of software maintenance, ongoing operation, and support. Various appliances (email, DNS, etc) are one popular implementation of SaaS.
Webmail, web hosting and other new "groupware" technologies (blogs, wikis, web-conferencing, etc) can he hosted on the local network cheaper and more reliably then on Google or Amazon and they enjoy tremendously better bandwidth due to usage of local network (which now is often 10Gbit/sec) instead of much slower (and in case of private networks more expensive) WAN. Web hosting providers emerged as a strong, growing industry that essentially pioneered the commercialization of SaaS model and convincingly proved its commercial viability. As Kismore Swaminathan aptly noted in
his response to Alastair McAulay�s article
.... as presently constituted, cloud computing may not be a panacea for every organization. The hardware, software and desktop clouds are mature enough for early adopters. Amazon, for example, can already point to more than 10 billion objects on its storage cloud; Salesforce.com generated more than $740 million in revenues for its 2008 fiscal year; and Google includes General Electric and Procter & Gamble among the users of its desktop cloud.
However, several issues must still be addressed and these involve three critical matters: where data resides, the security of software and data against malicious attacks, and performance requirements.
Hillary Clinton private emails server scandal and Podesta email hack provides additional interesting view of the problems of cloud computing,
Actually the fact that General Electric and Procter & Gamble use Google arises strong suspicion about quality of high IT management in those organizations. IMHO this is too risky gamble to play for any competent IT architect. For a large company IT costs already are reduced to around 1% or less, so there are no big saving in going further in cost saving direction. But there are definitely huge risks, as as some point quantity of cost cutting turns in real quality of service issue.
I would argue that "in the cloud" paradise looks more like software demo from a popular anecdote about the difference between paradise and hell ;-). It turns out that for the last five years there are several competing technologies such as usage of virtual appliances and autonomous datacenters or as they are sometimes called "datacenter in the box".
Usage of a local network eliminates the main problem of keeping all your data "in the cloud": possibility of network outages and slowdowns. In this case all local services will continue to function, while in the "pure" cloud services are dead. From the end user perspective, it doesn�t make a difference if a server glitch is caused by a web host or a software service provider. Your data is in the cloud, and not on your local PC. If the cloud evaporates you can�t get to your data. This iss well known to Google users. If the service or site goes down, you can�t get to your data and have nobody to contact. And even if you have somebody to contact as this is a different organization, they have their own priorities and challenges.
While software as a service might represent a licensing savings, there might be a better way to achieve the middle ground and lot to lose all advantages of local storage of data. For example by using some kind of P2P infrastructure automatically synchronized so that you can view/edit data locally. Data should not be held hostage until the user can get back to the cloud. In this sense Microsoft's Microsoft Live Mesh might be a step in right direction as it provides useful middle grown by synchronizing data across multiple computers belonging to a users (home/office or office1/office2, etc).
But the key problem with "in the cloud" delivery of services is that only some services, for example Web sites and email, as well as others with limited I/O (both ways) are suitable for this deployment mode. Attempting to do company wide videoconference via cloud is a very risky proposition.
That does not means that the list of services that are OK for centralization and can benefit from it is short. Among "limited I/O" services we can mention payroll and enterprise benefits services -- another area where "in the cloud" computing definitely makes sense. But most I/O intensive enterprise processing like file sharing is more efficiently done on a local level. That includes most desktop Office related tasks, ERP tasks to name just a few. They are more difficult and more expensive to move into the cloud and economic justification for such a move is open for review. So in a way it is a complementary technology to the local datacenter not an alternative one. Moreover "in the cloud" approach can be implemented on a local level over LAN instead of WAN ("mini cloud" or "cloud in the box").
Actually cloud-based services itself are not that cheap so cost savings in switching to the "in the cloud" service provider for large organizations are typically exaggerated and might even be illusory.
The top rated providers such as Amazon are mainly interesting, if you experience substantial, unpredictable peaks in your workloads and or bandwidth use. For stable, consistent workloads you usually end paying too much . Spectrum of available services is limited and outside running your own virtual servers it is difficult to replace services provided by a real datacenter. May be except small startups. The most commercially viable part is represented by Web hosting and rack hosting providers. For web hosting providers the advantages are quickly diminishing with the complexity of website, though. Among Amazon Web services only S3 storage currently can be called a successful, viable service. Microsoft Live Mesh is mostly a data synchronization service. It presuppose existence of local computers and initially provides syncing files between multiple instances of local computers belonging to the same user. This is an important service, but this not a pure "in the cloud" provider model. It is a more realistic "mixed" approach.
Again, with the current prices of servers and network equipment, most existing services are not cheap and became really expensive as you scale them up. We will discuss this issue later.
All-in-all it is not very clear what market share "in the cloud" services deserve and but it is clear that "in the cloud" service providers can't fit all enterprise needs. Currently, there are a lot more places to run software than there used to be, thanks to the proliferation of powerful laptops, mobile phones like iPhone/Android phones, tablets, etc. Actually smartphones represent another interesting development that runs in the direction opposite to the move to the "in the cloud" computing. It is a powerful local device with wireless network connectivity. The key here is substantial processing power and storage capacity available on the device which is increasing with each new model.
From social standpoint cloud provider services are also problematic. Service providers mind their own interests first. also large service provider is the same bureaucratic monster as a typical datacenter that share with the latter all the spectrum of Dilbertalization problems. Large customers experience "vendor lock-in" working with service providers as it involves significant effort to adapt on both sides. So walk-out is a less viable option that one can assume on pure theoretical backgrounds. It is also possible that outsourcing to "software service providers" like any outsourcing can be used as a negotiating tool (aka "method of blackmail"), to force wage concessions from workers, tax breaks from governments, etc., even when the actual savings would be small, or even when they are negative and moving
makes no business sense whatsoever.
Promoters of remote outsourced datacenter, such as Nicholas Carr usually ignore the availability and the cost bandwidth. Think Netflix and all conflicts it is fighting with the local cable internet providers. We can't assume that free Internet connectivity is adequate for all business purposes. such an assumption is corrected called "bandwidth communism". Yes, fiber-optic changed WAN landscape making remote services more viable and Internet tremendously more powerful. But the devil is in details. For example file sharing for a large company over WAN is still bad idea as public bandwidth is insufficient and private is costly. Also any enterprise making bet of 24x7 availability of public bandwidth for vital corporate services looks slightly suicidal because of the "tragedy of commons" problem, which already demonstrated itself in repressions against P2P enthusiasts by large ISPs. All-in-all this "grand utility computing" vision ("bandwidth communism") is problematic because somebody needs to pay for all this expensive infrastructure.
Fiber networks increased both Internet and LAN bandwidth substantially and revitalized distributed computing. But there is a big difference whether you distribute over LAN or WAN. The latter is much tougher case. With all the tremendous progress of Internet available bandwidth does not increase as quickly as computing power. Nowhere close, and it never has. If anything, due to increased scrutiny and "shaping" by ISPs (they are not a charity and need to be profitable) bandwidth "per user" might recently start decreasing as such resource hogs as YouTube and video programming distribution services (movies on demand) are becoming more and more prominent. Ability of video streams and P2P services to clog the Net in the most inopportune moment now is well established fact and is a real curse for university networks.
For i/o intensive tasks, unless you pay for the quality of service, "in the cloud" computing model stands on a very shaky ground. Reliable 24x7 bandwidth cannot be free for all users in all circumstances and for all protocols. Substantial amount of traffic with the remote datacenter is not that easy to transmit reliably and with minimal latency via public channels in rush hours. But buying private links to remote datacenters can be extremely expensive: for mid-side companies it is usually as expensive as keeping everything in house. For multinationals it is more expensive, so only "other considerations" (like "jurisdiction over the data") can sustain the centralization wave to the large remote datacenters. For many multinationals SOX was the last straw that made move of datacenters out of the USA desirable, costs be damned. Now the shadow of NSA serves as keeping this scare alive and well. Cost of private high speed links limits cost efficiency of the "in the cloud" approach to any service where disruptions or low bandwidth in certain times of the day cannot lead to substantial monetary losses. For critical business services such as ERP public data channel can be too brittle.
But it is fair to state that the situation is different for different services. For example for SMTP mail outsourcers like Google/Postini, this problem is not relevant due to the properties of the SMTP protocol: they can communicate via regular Internet. The same is true to DNS services providers, webmail and instant messaging. CRM is also pretty close. But for ERP, file sharing and WAN based backup the situation is very different: providing high speed networking services over WAN is a very challenging engineering task to say the least. The cost of bandwidth also puts natural limits on service providers growth as local networks are usually much cheaper and much faster. Achieving 1Gbit/s speed on LAN is extremely cheap (even laptops now have 1Gbit adapters) while it is quite expensive over WAN. There are also other limiting factors:
- The possibility of creation of local LAN backbone with speeds higher the 1 Gbit/s. 10Gbit/s backbones are becoming more and more common.
- Limited bandwidth at the point of connection of provider to the Internet. Every provider is connected to the Internet via a pipe and that pipe is only so big. For example OC-1 and OC-3 have their upper limit of 51.84Mbit/s and 155.2 Mbit/s correspondingly. Funny the upper speed of OC-3 (which is pretty expensive) is only slightly higher that 100Mbit/s which long ago became the lowest common denominator for LANs. Large service providers typically use OC-46 with speed up to 2488.32 Mbit/s which is similar to the speed of gigabit Ethernet. 10 Gigabit Ethernet is the fastest commonly available network standard for LANs. It is still emerging technology with only 1 million ports shipped in 2007 but it is quickly growing in importance. It might be eventually used in modified form for WANs too. Anyway as WAN bandwidth is limited and shared between multiple customers the spike in activity of one customer might negatively affect others. Networking problems at the provider level affect all its customers and recovery period might lead to additional spikes of activity.
- Local vs. remote storage of data. Recent enterprise level hardrives (Cheetah 15K) have speed up to 164 MB/sec (Megabytes, not megabits). From the speed and cost point of view the ability to keep data/programs local is a big technological advantage. For I/O intensive applications it might be that the only viable role for remote providers is synchronization with local data ;-). Example of this approach is Microsoft's Live Mesh
- Interdependence of customers on the transport level. This is jut another incarnation of "tragedy of commons" problem. Bandwidth hogs like game, P2P, music and video enthusiasts do not care a dime about your SLA and can easily put a company that uses public links into disadvantage any time of the day if and when something new and exiting like a new HD movie was released. Also providers are not willing to sacrifice their revenue to accommodate "free-riders.": as soon as usage of bandwidth cuts into profits it is punishable and no amount of rhetoric about "Internet freedom" and "Net neutrality" can change that. That means that enterprise customers relying on public bandwidth can suffer from the effort of providers to manage free-riding. That means the corporation which moved services to the cloud competes with various bandwidth hogs who do not want to scarifies any ground and ready to go quite far to satisfy their real or perceived needs. My university experience suggest that corporate users can suffer from Internet clogging in the form of sluggish download speeds, slow response times and frustration with i/o intensive services that become much less useful and/or enjoyable. See for example Time Warner Cable Vs. The Horde.
- Competition for the resources at remote datacenter level. For any successful service providing all the necessary bandwidth is costly and cuts into margins. Recently Amazon faced the situation when bandwidth required for its Elastic Compute Cloud (EC2) proved to be higher then by all of Amazon.com�s global websites combined. You can read between lines how that affect profitability:
Adoption of Amazon Elastic Compute Cloud (EC2) and Amazon Simple Storage Service (S3) continues to grow. As an indicator of adoption, bandwidth utilized by these services in fourth quarter 2007 was even greater than bandwidth utilized in the same period by all of Amazon.com�s global websites combined.
Web services providers which offer customers unlimited bandwidth are banking on the fact that the majority of their customers will not use much of their bandwidth. This is essentially a marketing trick. As soon as you exceed a fraction of what is promised they may well kick you out. People who tried to implement software , mp3 or video sharing services on low cost ISP accounts realized that very soon. See for example references that I collected under "Unlimited bandwidth myth". Web neutrality does not mean the tragedy of commons is not applicable. As Bernardo A. Huberman, Rajan M. Lukose noted:
Because the Internet is a public good and its numerous users are not charged in proportion to their use, it appears rational for individuals to consume bandwidth greedily while thinking that their actions have little effect on the overall performance of the Internet. Because every individual can reason this way, the whole Internet's performance can degrade considerably, which makes everyone worse off. An analysis of the congestions created by such dilemmas predicts that they are intermittent in nature with definite statistical properties leading to short-lived spikes in congestion. Internet latencies were measured over a wide range of conditions and locations and were found to confirm these predictions, thus providing a possible microscopic mechanism for the observed intermittent congestions of the Internet.
So a company which will try to implement Web based streaming of say corporate video conference via cloud is up to nasty surprises unless it paid "arm and leg" for dedicated lines to its headquarters and other major locations (which make the whole idea much less attractive in comparison with the local datacenter). The ability to stream video of any considerable quality in real-time between two (or more!) arbitrary points in the network is not really something that can be easily done over the current Internet.
The main point to make is that a reliable WAN network connectivity cost a lot of money is difficult to achieve. This problem is unavoidable if your major components are "in the cloud" (in WAN). Also in the "free internet" enterprises are starting to compete for bandwidth with streaming media (films over Internet). The latter proved to be a huge resource hog and quality of a typical Internet connection now fluctuates widely during the day. That means that in order to achieve respectable quality of service for bandwidth intensive applications enterprises need to buy dedicated WAN connections. That is a very expensive habit to say the least. In typical for multinationals moves, say, relocation of SAP/R3 instance from USA to Europe (just from one continent to another) to achieve reasonable latency for requests coming from the USA is not that easy and definitely not cheap. The cost of high bandwidth transatlantic connection is the major part of additional costs and eats all savings from the centralization. The same effect is true about any WAN connection: reliable high-bandwidth WAN connections are expensive. Moreover the reliability needs to be carefully monitored and that also cost money as anybody who was responsible for the company WAN SLA can attest.
| ||Centralisation, or centralization (see spelling differences), is the process by which the activities of an organization, particularly those regarding decision-making, become concentrated within a particular location and/or group. |
There is a shadow of mainframes over the whole idea of "in the cloud" software services providers. Excessive centralization has its own perils, perils well known from the days of IBM360/370 and "glass datacenters". While at the beginning growth of "cloud services" providers increase profit margins and may even increase the quality of the service eventually each organization hits "size" wall.
That's why businesses tend to perceive such providers as inflexible, and, due to natural pre-occupation with profit margins, hostile to their business needs: exactly like their perceived "grass datacenter" in not so distant past. So much for the improvement of the business model in comparison with traditional local datacenters, as problematic as they are (and I would be the last to deny that they have their own set of problems).
As one reader of Calculated Risk blog noted in his comments to the post
Popular Google Product Suffers Major Disruption
They started employing more MBAs at Google? Seriously, any creative company that "grows up" and starts taking advice from bean counters/ entrail readers and sociopaths is doomed.
Microsoft's greatest strength was that its founders never lost control.. so even though their products were inferior (at least initially), their competitors took advice from MBAs and f**ked themselves up, leaving MS as the only survivor.
It's very hard to scale service-based tech firm and keep the mojo that made the startup successful in the first place, especially via acquisitions or employing 'professional managers' to operate the company. Basically I think the story is simple -- Parkinson's law -- bureaucracies naturally grow without limit. That includes the management of large "cloud-services" providers including Google. Excessive layers of middle managers and affiliated cronies ("shadow secretaries") count in the denominator of labor productivity. Everyone knows that many middle managers (often called PHBs) are mainly "inventing" work for themselves and other middle managers writing memos and calling meetings and stuff. Middle management is all about sustaining internal information flow; technology makes good middle management more efficient since it enables better information flow but it makes bad middle manager even more harmful as they "invent" useless information flow (spam) and block useful information in order to survive.
Issues of quality, loyalty, knowledge of the business are automatically surfaced and as a result customers suffer. Mainframe-style utility model encourages excessive bureaucratization with rigid procedures, stupid customer service and power concentrated at the top. That means that the issues of sociopathic leadership and sycophant managers replacing founders is even more acute then in regular IT firms. Corporate predators prefer large companies. As a result demoralization surface and IT personnel, that is cubicle serfs, now spend a large fraction of their time in the office, surfing the internet.
Contrary to simplistic description and assumptions typical for writers like Carr mainframe warts are very difficult, if not impossible to overcome. And they can be amplified by low cost of free services with no reliability guarantee. Disappearance of data usually is not covered. There is a danger of relying of semi-free (advertisement supported) services too much.
For example anybody who used low cost Web-hosting provider can attest that interests of providers run contrary to the interests of advanced users and as such often stifle advanced technology adoption even if they are supported by a particular provider because they, for example, might increase the load on the server. Also provision of the adequate bandwidth for multiple users (and decent responses times) can be a shaky area. Especially during rush period like 8-11 EST. Typically customer service is far from being responsive to any complains.
Security is another important (and costly to resolve) challenge: break-in into a large provider affects all its customers. There were several high profile break-ins into large Web hosting providers during the last two years, so this is not a theoretical threat. Claiming that Web provider are a total solution for any organization is like saying that just because the Big 4 accounting firms (with their the army of accountants, tax specialists and so on) exist, organizations can do away with internal accountants altogether. The hybrid platforms, such as Saleforce.com's application upgrades and quality of service issue still are of major concern.
Relying on advertisement is another mine field. Many people hesitate to send anything important to a Gmail address, knowing that the mail will be scanned (by what is an advertising company) in transit.
Still there is a lot of disappointments with this model as exemplified with the following characterization of "cloud-based" services and outsourcing:
'We thought that we are like a farmer shooing a fat rabbit, but it turned out that the rabbit is shooing at us."
This quote suggests that providers of such services as any outsourcers in the future might have difficulties to shake money loose from the organizations, as customers discover that the interests are highly divergent. IBM already discovered that this is an inherent limit of their push to the "'service oriented organization". Recently they even experienced a backlash.
Because we need to bridge interests of two or more different organization, there are several significant problems in interaction with "in the cloud" providers. Among them
- Problem of loyalty. It cuts both ways. First of all in case you use external providers loyalty of staff disappears and for any complex service you face "all or nothing situation": If service works everything is wonderful, but if it does not troubleshooting is extremely difficult and fixing the problem is almost impossible even if you understands what is happening -- infrastructure belongs to other organization. Anybody who used Web hosting (the most successful example of such services) can attest that this is a wonderful service as long as it works. But if you have a problem you have a big problem: local staff has no loyalty to the particular organization and competition does not work as another provider can be as bad or even worse; so switching brings you nothing but additional headache and losses. Even elementary problem can take several months to be resolved and I am not joking I experience it myself. Oversubscription which leads to highly loaded servers and insufficient network bandwidth is another common problem. There are also problems related to the "race to the bottom" in such services: the main differentiator becomes price and to attract new customers Web providers often make claims that are untrue (unlimited bandwidth is one typical example). As a result naive customers who believe in such claims are burned.
- Forced upgrades. In case of external providers that platform is owned by the provider and if his financial or technological interests dictate that upgrade is necessary it will be done. That means that customers have to adapt or leave. And upgrade for a provider with large number of customers can be huge mess that cost clients dearly. I experienced this situation myself and can attest that the level of frustration involved is substantial and the mess can easily last several months.
Compatibility problems. As provider uses specific technology the level of interoperability of this technology with other important for the company technologies is not under the control of the user of such services. It can lead to significant costs due to luck of interoperability. In the simplest example lack of interoperability with Microsoft Office is a serious problem for Sun which uses Open Office (Star Office to be exact).
The loss of operational flexibility When switch to "in the cloud" provider is done on cost grounds alone, that usually creates new (and initially mostly unrecognized) dependencies that deprive the company from much of the operational flexibility. Typically security departments are direct beneficiary as security-related spending tend to grow dramatically as a defensive reaction of the organism. The main consequence of the bureaucratization is the loss of flexibility, and sooner or later this lack of flexibility can come back and haunt the company.
In the recent interview Bill Gates noted that "The IT systems are your brain. If you take your brain and outsource it then any adaptability you want (becomes) a contract negotiation". After you negotiated the contact with the "in the cloud" provider any flexibility you used to have is lost as every change explicitly or implicitly become a change of the contact negotiation. Moreover, if the company lost its key IT staff and key architectural issues are decided by outsourcers, then essentially the company becomes a hostage of outsourcers as it no longer has brain power to access the options and the architecture (and thus the costs) are by-and-large controlled by forces outside your control. That is much more serious risk that many PHB assume: the line between architectural decisions and implementation decisions is pretty fuzzy. There is also associated brain-drain risk � if you outsource important functions, you irrevocable erode the capabilities within your firm. When problems arise, internal IT staff can often resolve the problem more quickly, with less bureaucratic overhead inherent when two corporations are involved.
The fist category of factors that lead to loss of flexibility is connected with additional standards, policies and procedures that are usually introduced in external service provider situation.
The other important aspect of the loss of remnants of competitive advantage, if any, as the service provider is now the place were the critical mass of know-how and talent pool reside. That somewhat reverses "client-provider" relationship: it is service provider who now can dictate the rules of the game and who is the only party who understands the precise nature of the tasks involved and can conceal this knowledge for his own advantage from the other party. That usually is helped by the demoralization of the remaining staff in the company
- Amplification of the management missteps.
Their first and most important desire of service provider is to keep the client, even if client's situation changed and no more lend itself easily to the services provided. That can lead to huge architectural blunders. Those things are not limited to offshoring but happened often with complex projects like ERP were external consultants are involved, especially in case big consultant firms are involved. Several large companies went out of business or were bought as a result. Among example we can list FoxMeyer, AT&T Wireless. Several companies were severely hurt (Herschel, etc). This is might actually include Compaq.
As for Compaq, the story is more complex. Dot-com bust hurt "value added" companies like HP and Compaq disproportionally and increased appetite for mergers and acquisitions activities. Dogged determination of Carly Fiorina about the merger (which served both for self-promotion and as a smokescreen for her own problems at struggling HP) and the ability of former Compaq boss, a certified public accountant Michael Capellas, to understand personal benefits from Carly Fiorina proposition proved to be a winning combination. Capellas who proved to be a "serial CEO", , was a big fan of SAP and that probably was a contributed factor is Compaq troubles. When he was appointed, he promised to turn around struggling Compaq in 180 days. Now we know that after just 84 he found a much better financial solution, at least for himself. A shortcut for a turnaround ;-). It is interesting to note that in 2000, based on iPAQ success, he declared[ ComputerWeekly.com,Feb 2000] :
"There will be an explosion in devices, all attached to the Web. We are going to simplify the PC radically." Capellas promises that future desktop computers will be easier to manage than today's machines.
Large companies have become very adept at establishing remote teams in places like India, Russia, etc. Due to their ability to offer higher wages and better environment they are able to attract local talent and run challenging research and development projects. Often, though, these outposts become disconnected from their parent companies because they fail to establish rapport with key participants in the central office. Foreign prophets are usually ignored. There is something fundamental in this "tragedy of local talent" and communication is still a problem even in the age of videoconferences. Without "warm-body" personal contacts it is difficult to build long-term trust based relationships.
Many organizations who thought outsourcing IT was the key to success miserably failed. Whose who did not failed lost competitive advantage, experienced the demoralizing effect of vendor lockdown and QOC hokey-pokey which just simply made no sense. Some in-sourced the IT back, some recreated it from scratch, are still in denial.
That means that client of service providers will be implicitly pushed to lowest common denominator and cannot utilize local expertise, even if such exists. They face "helpdesk level" people and instead of benefiting from specialized provider are often proposed wrong solutions to misdiagnosed problems. my experience with WEB providers suggests that trivial problems like an error in DNS record or wrong permissions can became real production issues.
Service providers can evolve and upgrade software independently of wishes of some or all of the customers. That means that customers who are not satisfied with the direction taken need either to adapt or abandon the service.
Public internet is unsuitable for handling large volume of transaction with stringent performance criteria. That means that it is dangerous to put databases at "in the cloud providers" : the more successful "in the cloud" providers are ( or if there just are too many P2P and or multiplayer videogames enthusiasts in the same subnet), the slower your WAN connection will be ("tragedy of commons").
Moreover, in comparison with LAN, WAN-based provision of software services is more complex system and as such is less reliable especially at bottlenecks (which are service provider "entry points" and associated infrastructure (DNS, routers, switches, etc). With WAN outage the situation can became a lot worse then when just when spreadsheets or MS Word documents suddenly are inaccessible on the local server due to LAN outage (but you can still download then into USB stick directly from the server and work with the local copy until network connectivity is restored, because your departmental file server is just several dozens of yards away and friendly administrator probably can help you to get to the data. In case of WAN there is no way to put a USB stick on the server or use other shortcut to avoid effects of network downtime: if WAN connection is down you are really down. Generally not only you can do nothing about the outage, its effects might be amplified by the fact that there are many customers affected. All you will get is the message like this:
The service is experiencing technical difficulties. We apologize for the inconvenience. Please try again later .
That means that in some cases the effect on organization of external outage might be such that the head of the person who enthusiastically recommended company to move "into the cloud" rolls down independent of his position, real or faked IQ and technical competence. Recently both Gmail and Amazon services experienced multiple outages. As Brad Stone noted in NYT:
There is plenty of disappointment to go around these days. Such technology stalwarts as Yahoo
and Research in Motion
, the company behind the BlackBerry, have all suffered embarrassing technical problems in the last few months.
About a month ago, a sudden surge of visitors to Mr. Payne�s site began asking about the normally impervious Amazon. That site was ultimately down for several hours over two business days, and Amazon, by some estimates, lost more than a million dollars an hour in sales.
The Web, like any technology or medium, has always been susceptible to unforeseen hiccups. Particularly in the early days of the Web, sites like eBay and Schwab.com regularly went dark.
But since fewer people used the Internet back then, the stakes were much lower. Now the Web is an irreplaceable part of daily life, and Internet companies have plans to make us even more dependent on it.
Companies like Google want us to store not just e-mail online but also spreadsheets, photo albums, sales data and nearly every other piece of personal and professional information. That data is supposed to be more accessible than information tucked away in the office computer or filing cabinet.
The problem is that this ideal requires Web services to be available around the clock � and even the Internet�s biggest companies sometimes have trouble making that happen.
Last holiday season, Yahoo�s system for Internet retailers, Yahoo Merchant Solutions, went dark for 14 hours, taking down thousands of e-commerce companies on one of the busiest shopping days of the year. In February, certain Amazon services that power the sites of many Web start-up companies had a day of intermittent failures, knocking many of those companies offline.
The causes of these problems range widely: it might be system upgrades with unintended consequences, human error (oops, wrong button) or even just old-fashioned electrical failures. Last month, an electrical explosion in a Houston data center of the Planet, a Web hosting company, knocked thousands of Web businesses off the Internet for up to five days.
�It was prolonged torture,� said Grant Burhans, a Web entrepreneur from Florida whose telecommunications- and real-estate-related Web sites were down for four days, costing him thousands of dollars in lost business.
I was actually surprised how much posts each "in the cloud" service outage generates and how significant were losses reported by some users; in addition to Official Gmail Blog one of the best places to track Gmail outages proved to be Twitter; there is also a site
http://downforeveryoneorjustme.com/ which provides a free check for frustrated Gmail and other "in the cloud" services users). Some reactions are pretty funny:
Thank you so much for coming back. I felt lonely without your constant, vigilant presence.
I was so relieved Gmail was down. I thought my employer decided to block it on my berry.
Gmail ruined my day. Damned cloud!
Gmail is really pissing me off today, which doesn't happen often. S'like having your first real fight in a relationship. Very disconcerting!
Gmail's back, but now my IndyChristian provider is down. And Facebook.com . Go figure.
Thank God I never jumped aboard the evil gmail ship!
To all Gmail outage whiners: if you want the right to bitch, don't use free email. Also, look up the definition of "BETA" sometime.
Very comforting to know that I wasn't the only one that had no Gmail for a bit there. I'm not alone in this world? Long but good day.
Man, gmail goes down for one moment and the whole world is up in arms :)
How did I miss the gmail outage? I feel so out of the loop!
the weather is hot, the a/c is b0rked, gmail was down this afternoon. expect to see four horsemen walk by any minute...
orry, completely missed the gmail outage because I was out biking and enjoying the amazing evening we're having here.
Having Gmail down about made me bang my head on the table. Oy.
oycotting gmail and getting behind the olympics
For Twitterers, Gmail going down is like JFK being shot, innit?
Ignorance IS bliss-so busy today I did not even know about the great gmail outage of '08.
The cloud of unreliability: It's not clear why anyone should be surprised that Gmail, Amazon.com's cloud .. http://tinyurl.com/649amj
Was thinking about going to back to Gmail from .Mac since MobileMe was down. Now I hear Gmail was down. Thinking about envelopes and stamps
so mad at gmail for 502-ing her earlier today. what did we do before google?
With Gmail having gone down and me not noticing, I am undiagnosing myself of Internet Addiction and OCD.
Maybe gmail is down to express solidarity with MobileMe?
As any self-respecting obsessive business e-mail checker could tell you, each outage is a real shock and fails on the most inopportune moment. In reality most email outages does not make users less productive, they just deprive them from their favorite tool of wasting own and other time and procrastination ;-)
Here is a short but pretty sobering list for 2008. It neither complete not contains the most important posts about particular outage.
[Jan 28, 2008] Gmail OUTAGE, relying on Google too much - TECH.BLORGE.com
Google services are usually rock solid reliable, but earlier today some Gmail users lost service for a couple of hours. That begs the question, are we relying on Google too much?
Gmail outages are peppered over the last few years and every time it creates a flurry of activity as businesses and individuals realize how much they rely on a single business and platform for all of their messaging needs.
The same concept is mirrored with search engine traffic, where many people don�t even realize there are search engines out there other than Google.
[Feb 16, 2008] Amazon explains its S3 outage Between the Lines ZDNet.com
Amazon has issued a statement that adds a little more clarity to its Web services outage on Friday.
Here�s Amazon�s explanation of the S3 outage, which wreaked havoc on startups and other enterprises relying on Amazon�s cloud.
Early this morning, at 3:30am PST, we started seeing elevated levels of authenticated requests from multiple users in one of our locations. While we carefully monitor our overall request volumes and these remained within normal ranges, we had not been monitoring the proportion of authenticated requests. Importantly, these cryptographic requests consume more resources per call than other request types.
Shortly before 4:00am PST, we began to see several other users significantly increase their volume of authenticated calls. The last of these pushed the authentication service over its maximum capacity before we could complete putting new capacity in place. In addition to processing authenticated requests, the authentication service also performs account validation on every request Amazon S3 handles. This caused Amazon S3 to be unable to process any requests in that location, beginning at 4:31am PST. By 6:48am PST, we had moved enough capacity online to resolve the issue.
As we said earlier today, though we�re proud of our uptime track record over the past two years with this service, any amount of downtime is unacceptable. As part of the post mortem for this event, we have identified a set of short-term actions as well as longer term improvements. We are taking immediate action on the following: (a) improving our monitoring of the proportion of authenticated requests; (b) further increasing our authentication service capacity; and (c) adding additional defensive measures around the authenticated calls. Additionally, we�ve begun work on a service health dashboard, and expect to release that shortly.
The Amazon Web Services Team
[Jun 9, 2008] Surviving gMail 502 outages SKFox
I use mint to see live stats for the traffic to skfox.com and Google Analytics for the long term statistics. I�ve seen a resurgence today of traffic to my two blog posts (in March and again in May) about my gMail account being down. The most common for me is a �Temporary Error (502)� and it would seem to be happening to other users too. I hate to be the bearer of bad news, but there isn�t a whole lot you can do about it. On the short side, most of the outages only last 30 minutes or so with the longest being 3.5 hours. It can be terribly frustrating, but there are a few things you can do to alleviate the pain.
- Use your email client like Outlook or Thunderbird to download your messages to your local machine with POP3 while keeping your gmail account unchanged. That way, even if gMail is inaccessible, you can get to your old email for reference. Of course, you have to do this once your account is up and running again.
- Visit the gMail Google group to at the least find others in your boat and get the latest updates.
Some users have reported success getting to their mail during outages by using any number of alternative links to gmail. Your mileage may vary, but here they are:
Today�s outage is a known issue, and I hope for all of your coming here from search engines, that it comes back up quickly for you.
[Jul 20, 2008] Amazon's S3 Storage Outage Busts Web Sites, Crashes iPhone Apps
What happened? Sometime this morning, Amazon's (AMZN) S3 storage service went down. Or, according to Amazon's official note, S3 is "experiencing elevated error rates." A lot of companies -- Twitter, 37signals, etc. -- rely on S3 to host static files like images, style sheets, etc. for their Web apps. So when S3 goes down, those sites lose some/most/all functionality, depending on what the company uses S3 for.
So how'd Amazon's storage service bork my iPhone? Tapulous relies on S3 for images like Twinkle user icons. And they must not have included a "plan B" in their code to handle an image server outage. So when S3 hiccuped, and Twinkle couldn't download images, the app crashed, taking me back to the iPhone home screen. (And, hours later, it's still crashing.)
[Aug 11, 2008] TheStar.com Business Google Gmail users having troubles
SEATTLE�Google Inc said Monday that many users of its Gmail service are having trouble accessing their online e-mail accounts due to a temporary outage in a contacts system used by Gmail.
The problems began at about 5 p.m., and a company spokesman said Google is starting to implement a fix for the problem right now.
"We are starting to roll out a fix now and hope to have the problem resolved as quickly as possible," said a Google spokesman.
[Aug 25, 2008] FOXNews.com - Amazon's Site Outage Raises Scalability Questions - Science News Science & Technology Technology News
Amazon.com has built a reputation for being an aggressive, take-no-prisoners kind of company, but it showed its more cooperative side this week.
Late last week, Internet traffic monitoring firm Keynote Systems issued a report that said many of the largest e-commerce players will face major load-handling challenges for the holidays if they don't make big changes by the fourth quarter.
However, after so many years of holiday shopping seasons, some in the industry scoffed that the majors could be caught so short.
To help out, Amazon (AMZN) generously knocked out its own servers for almost 2 hours on Aug. 21.
OK, OK. Amazon probably didn't crash just to help Keynote make a point. But its timing was impeccable nonetheless.
It's interesting to note that within a few days this month, three of the industry's most powerful retail forces � Wal-Mart (WMT), Amazon and Google (GOOG) � all suffered problems that, one way or the other, can be classified as scalability-related.
While there is nothing strange about such outages as large distributed datacenters are notoriously difficult to operate, please note that in case the organization uses both Amazon and Gmail they have multiple noticeable outages during the first half of 2008. At the same time it is impossible to deny that such outages definitely make fragility of in the cloud model visible even for the most "pro-clouds" observer such as Mr. Carr although I doubt that he'll reproduce the statistics listed above in his blog.
But the most important difference between "in the cloud" services and local services is not duration or frequency of the outages. The most important is the level and quality of information available about the situation. In case of local services all information about the situation is readily available and thus some countermeasures can be taken. In military world such difference is of paramount important. IT is not the different in this respect from military world. In case of "in the cloud" the amount and quality of information about the outage are very low; services customers are essentially in the dark. Services just abruptly seizes to exists and then magically comes back. This lack of information has its own costs.
|The most important difference between "in the cloud" services and local services is not duration or frequency of the outages. The most important is the level and quality of information available about the situation and level of loyalty of personnel involved. In case of local services all information about the situation is readily available and thus some countermeasures can be taken. In military world such difference is of paramount important. IT is not the different in this respect from military world. |
Virtualization generally magnifies failures. In the physical world, a server failure typically would mean that backup servers would step in to prevent downtime. In the virtual world, depending on the number of virtual machines residing on a physical box, a hardware failure could impact multiple virtual servers and the applications they host. As a result failures have a much larger impact, effecting multiple applications and multiple users. Little fires turn into big fires. Virtualization might also increase two other problems:
- Performance-related problems. Companies ensure top performance of critical applications using powerful dedicated servers, network and storage resources for those applications, segmenting them from other traffic to ensure they get the resources they need. "In the cloud" provider model of solving those problems is based on allocation of additional resources on demand as his goal is to minimize cost of infrastructure. That means that at any given time, performance of an application could degrade, perhaps not to a failure, but to a crawl making users less productive or paralyzed.
- Troubleshooting complexity. Server problems in the past could be limited to one box, but now the problem can be related to hypervisor, not the server per se or move with the application instance from on virtual machine to another. If problem existed on one server, but disappears after application is moved to another and later reappears the question arise: "Is the problem fixed?" Often temporary disappearance of the problem lull the staff into a vicious activity cycle and they start to transfer the problem around from virtual server to virtual server instead of trying to solve it
All-in-all the more a particular company relies on "in the cloud computing", the more severe will be effect of the outages. And in comparison with traditional local LAN-based "physical server" deployment there are several sources of them:
- In the cloud provider outages: upgrades, equipment replacements, DNS blunders (a very popular type of blunder ;-), blunders of personnel, problems connected with the personnel incompetence, etc.
- Links from in the cloud provider to ISP
- Problems in ISP (again here there are issues of upgrades, blunders of personnel, physical damages, DNS problems, etc)
- Problems in the link from ISP to the company
- Virtualization-related problems, if it is used by "in the cloud" provider.
The first thing you notice during large outage of "in the cloud" service provider is that customer service is overloaded and you need to wait a long time to get to the human voice. There are usually just too many angry and frustrated customers before you. Also bad is that the nature of the problem and the estimate of the time needed to resolve is usually are not communicated, keeping you in the dark. Moreover even after you get to a support person the information you get will be sketchy. Here is one example:
I was working on a client webinar using WebEx Event Center. I tried to log onto our site and got a cryptic message saying the site was unavailable. Whammo. I got on the line to technical support and waited my turn in queue, along with what must have been quite a few other panicked or angry customers. My support person was surprisingly calm and good natured, given the call volume they must have been processing. He confirmed that there were network problems on their side and said that while he didn't know details of the problem, he was certain it would be all right soon. I had a little time before our event, so I agreed to try again later.
Sure enough, our site came up, but on a backup network. This wasn't completely clean, as one of our scheduled future events had disappeared from our registration page, meaning people couldn't sign up. But at least we could carry on with today's show. Unfortunately, performance was quite a bit below normal standards, with slides and annotations taking inconsistent and sometimes very long time lags to appear on participants' computers.
After the event, strange behavior continued, with an inability to access post-event reports through the WebEx administrator interface. I called technical support again and got agreement that it certainly was strange behavior and we should all hope it would correct itself once the system was back to normal again. Whenever that might be.
Now, I'm not singling out WebEx for any particular acrimonious treatment here. I happened to be using them when this problem occurred. Any provider can have a similar problem. At least WebEx had a backup network plan in place and we were technically able to carry on with our scheduled event. But it's worth noting that there is a frustrating sense of powerlessness while a problem like this is going on. You can't prod anybody for more details, because you don't have access to the people trying to fix the problem as you might if a similar situation occurred with your own business network. You can't get much in the way of progress updates. And you can't put your own backup plan into effect.
One interesting difference in "horror stories" between local and "in the cloud" datacenter was aptly outlined in the following
Just after midnight on Jan. 22, 2006, Con Edison began telling the Internet that it was Martha Stewart. That is, Con Edison erroneously began sending out routing information to redirect Internet traffic intended for Martha Stewart Living Omnimedia to its own servers.
The publishing company was a Con Edison customer. So were other businesses and Internet providers whose routing information Con Edison hijacked at the same time. The result was a mess that wasn't completely cleared up for 18 hours � and some businesses were offline for most of that time.
But not Martha Stewart, whose CIO, Mike Plonski, wrote to me to clarify what happened at his company.
Plonski's secret sauce? No big secret � just network monitoring and redundancy.
Plonski said: "While one of the Internet connections at our corporate offices was impacted by the ConEd issue you describe, we, as a company, are smart enough to have employed redundancy, both by location and carrier, for our network operations. As a result, during this time frame, we simply flipped all of our Internet traffic to run over our secondary line until ConEd resolved their issue."
OK, it was a little more complicated than that. Plonski said his staff spotted the problem through routine network monitoring. There was clearly something wrong with network traffic coming to the corporate office over the Con Edison line. Thanks to the monitoring, the company knew about the problem about 30 minutes after it started.
Because of the type of outage, an IT staffer had to connect and manually switch over to a redundant line. That took another hour.
Total time for the outage: about 90 minutes in the wee hours of a Sunday morning. Total impact: minimal.
An outage? Yes. A knockout? No way. And handling the problem didn't require rocket science � just monitoring, redundancy and sharp IT staff work.
Repeat after me: [in case of local datacenters "handling the problem didn't require rocket science � just monitoring, redundancy and sharp IT staff work."
While individual "in the cloud" service provider can do a decent job in providing the required functionality, the problem arise when you need to use several such providers and issues of interoperability arise.
The lack of interoperability between different SaaS applications is one of the most well known and, at the same time, largest problems with SaaS. In a way companies who are using several different providers spending way too much money with the side effect of creating a problematic, cumbersome IT architecture that lack flexibility and efficiency and as such is inferior to the integrated datacenter architecture. The focus of vendors is on the adaptation of the SaaS model to enterprise requirements (enterprise-level integration), but there is a growing need for vendor-to-vendor integration. How and when those needs will be addressed remains to be seen.
In a way SaaS application emphasize too much GUI portion of functionality of application at the expense of the ability of smoothly exchange data with other, often similar applications.
The emergence of a several classes of enterprise SaaS (for email, CRM, supply chain management, benefits and other HR-related applications, etc ) creates problems of similar or identical data existing in various formats in different providers and their synchronization and timely updates. It providers does not share common "hardware cloud" we have a huge problems as not only protocols of data exchange, but reliability and security issues are pretty complex.
Also the existing interoperability can be broken anytime by software updates by one of several for existing "in the cloud" providers.
- 20181113 : GridFTP : User s Guide ( toolkit.globus.org )
- 20181112 : Edge Computing vs. Cloud Computing What's the Difference by Andy Patrizio , ( Jan 23, 2018 , www.datamation.com )
- 20181109 : Cloud-hosted date must be accessed by users over existing WAN which creates performance issues due to bandwidth and latency constraints ( Nov 09, 2018 , www.eiseverywhere.com )
- 20181109 : Troubleshoot WAN Performance Issues SD Wan Experts by Steve Garson ( Feb 08, 2013 , www.sd-wan-experts.com )
- 20181109 : Storage in private clouds ( Nov 09, 2018 , www.redhat.com )
- 20181109 : Cloud Computing vs Edge Computing Which Will Prevail ( Nov 09, 2018 , www.lannerinc.com )
- 20181109 : What is Hybrid Cloud Computing ( Nov 09, 2018 , www.dummies.com )
- 20181109 : Why Micro Data Centers Deliver Good Things in Small Packages by Calvin Hennick ( Nov 09, 2018 , solutions.cdw.com )
- 20181109 : Solving Office 365 and SaaS Performance Issues with SD-WAN ( Nov 09, 2018 , www.brighttalk.com )
- 20181109 : Make sense of edge computing vs. cloud computing ( Nov 09, 2018 , www.infoworld.com )
- 20181108 : GT 6.0 GridFTP ( Nov 08, 2018 , toolkit.globus.org )
- 20181108 : globus-gridftp-server-control-6.2-1.el7.x86_64.rpm ( Nov 08, 2018 , centos.pkgs.org )
- 20181108 : 9 Aspera Sync Alternatives Top Best Alternatives ( Nov 08, 2018 , www.topbestalternatives.com )
- 20181103 : Technology is dominated by two types of people; those who understand what they don t manage; and those who manage what they don t understand � ARCHIBALD PUTT ( PUTTS LAW ) ( Nov 03, 2018 , brummieruss.wordpress.com )
- 20181030 : Cloud has a rich future, but I didn't even know Red Hat had any presence there, let alone $35 billion worth. ( Oct 30, 2018 , arstechnica.com )
- 20181027 : We are effectively back to the centralized systems ("clouds" controlled by Amazon (CIA), Google (NSA)). The peripheral components of autonomous networked systems are akin to sensory devices with heavy computational lifting in the (local) regime controlled servers. ( Oct 27, 2018 , www.moonofalabama.org )
- 20170916 : Google Drive Faces Outage, Users Report ( Sep 16, 2017 , tech.slashdot.org )
- 20170702 : The Details About the CIAs Deal With Amazon by Frank Konkel ( Jul 17, 2014 , www.theatlantic.com )
- 20170609 : Amazon's S3 web-based storage service is experiencing widespread issues on Feb 28 2017 ( Jun 09, 2017 , techcrunch.com )
- 20170401 : Amazon Web Services outage causes widespread internet problems ( Apr 01, 2017 , www.cbsnews.com )
- 20170401 : After Amazon outage, HealthExpense worries about cloud lock-in by Maria Korolov ( Apr 01, 2017 , www.networkworld.com )
- 20170303 : Do You Replace Your Server Or Go To The Cloud The Answer May Surprise You ( Mar 03, 2017 , www.forbes.com )
- 20170112 : Digitopoly Congestion on the Last Mile ( Jan 12, 2017 , www.digitopoly.org )
- 20151109 : Thoughts on the Amazon outage ( nickgeoghegan.net )
- 20151109 : MAT Monitoring and Administration Tool ( MAT Monitoring and Administration Tool, )
- 20151109 : Unix SysAdm Resources Automated Unix SysMgmt Software ( Unix SysAdm Resources Automated Unix SysMgmt Software , )
"... file:///path/to/my/file ..."
"... gsiftp://hostname/path/to/remote/file ..."
"... third party transfer ..."
Table of Contents
1. Introduction The GridFTP User's Guide provides general end user-oriented information. 2. Usage scenarios 2.1. Basic procedure for using GridFTP (globus-url-copy) If you just want the "rules of thumb" on getting started (without all the details), the following options using
- 1. Introduction
- 2. Usage scenarios
- 2.1. Basic procedure for using GridFTP (globus-url-copy)
- 2.2. Accessing data in...
- 3. Command line tools
- 4. Graphical user interfaces
- 4.1. Globus GridFTP GUI
- 4.2. UberFTP
- 5. Security Considerations
- 5.1. Two ways to configure your server
- 5.2. New authentication options
- 5.3. Firewall requirements
- 6. Troubleshooting
- 6.1. Establish control channel connection
- 6.2. Try running globus-url-copy
- 6.3. If your server starts...
- 7. Usage statistics collection by the Globus Alliance
globus-url-copy will normally give acceptable performance:
globus-url-copy -vb -tcp-bs 2097152 -p 4 source_url destination_url
The source/destination URLs will normally be one of the following:
file:///path/to/my/file if you are accessing a file on a file system accessible by the host on which you are running your client .
gsiftp://hostname/path/to/remote/file if you are accessing a file from a GridFTP server .
2.1.1. Putting files One of the most basic tasks in GridFTP is to "put" files, i.e., moving a file from your file system to the server. So for example, if you want to move the file
/tmp/foo from a file system accessible to the host on which you are running your client to a file name
/tmp/bar on a host named
remote.machine.my.edu running a GridFTP server, you would use this command:
globus-url-copy -vb -tcp-bs 2097152 -p 4 file:///tmp/foo gsiftp://remote.machine.my.edu/tmp/bar
| ||Note |
|In theory, |
remote.machine.my.edu could be the same host as the one on which you are running your client, but that is normally only done in testing situations.
2.1.2. Getting files A get, i.e, moving a file from a server to your file system, would just reverse the source and destination URLs:
| ||Tip |
file: always refers to your file system.
globus-url-copy -vb -tcp-bs 2097152 -p 4 gsiftp://remote.machine.my.edu/tmp/bar file:///tmp/foo
2.1.3. Third party transfers Finally, if you want to move a file between two GridFTP servers (a third party transfer ), both URLs would use
gsiftp: as the protocol:
globus-url-copy -vb -tcp-bs 2097152 -p 4 gsiftp://other.machine.my.edu/tmp/foo gsiftp://remote.machine.my.edu/tmp/bar
2.1.4. For more information If you want more information and details on URLs and the command line options , the Key Concepts Guide gives basic definitions and an overview of the GridFTP protocol as well as our implementation of it. 2.2. Accessing data in...
2.2.1. Accessing data in a non-POSIX file data source that has a POSIX interface If you want to access data in a non-POSIX file data source that has a POSIX interface, the standard server will do just fine. Just make sure it is really POSIX-like (out of order writes, contiguous byte writes, etc).
2.2.2. Accessing data in HPSS The following information is helpful if you want to use GridFTP to access data in HPSS. Architecturally, the Globus GridFTP server can be divided into 3 modules:
In the GT4.0.x implementation, the data transform module and the DSI have been merged, although we plan to have separate, chainable, data transform modules in the future.
- the GridFTP protocol module,
- the (optional) data transform module, and
- the Data Storage Interface (DSI).
| ||Note |
|This architecture does NOT apply to the WU-FTPD implementation (GT3.2.1 and lower). |
126.96.36.199. GridFTP Protocol Module The GridFTP protocol module is the module that reads and writes to the network and implements the GridFTP protocol. This module should not need to be modified since to do so would make the server non-protocol compliant, and unable to communicate with other servers.
188.8.131.52. Data Transform Functionality The data transform functionality is invoked by using the ERET (extended retrieve) and ESTO (extended store) commands. It is seldom used and bears careful consideration before it is implemented, but in the right circumstances can be very useful. In theory, any computation could be invoked this way, but it was primarily intended for cases where some simple pre-processing (such as a partial get or sub-sampling) can greatly reduce the network load. The disadvantage to this is that you remove any real option for planning, brokering, etc., and any significant computation could adversely affect the data transfer performance. Note that the client must also support the ESTO/ERET functionality as well.
184.108.40.206. Data Storage Interface (DSI) / Data Transform module The Data Storage Interface (DSI) / Data Transform module knows how to read and write to the "local" storage system and can optionally transform the data. We put local in quotes because in a complicated storage system, the storage may not be directly attached, but for performance reasons, it should be relatively close (for instance on the same LAN). The interface consists of functions to be implemented such as send (get), receive (put), command (simple commands that simply succeed or fail like mkdir), etc.. Once these functions have been implemented for a specific storage system, a client should not need to know or care what is actually providing the data. The server can either be configured specifically with a specific DSI, i.e., it knows how to interact with a single class of storage system, or one particularly useful function for the ESTO/ERET functionality mentioned above is to load and configure a DSI on the fly.
220.127.116.11. HPSS info Last Update: August 2005 Working with Los Alamos National Laboratory and the High Performance Storage System (HPSS) collaboration ( http://www.hpss-collaboration.org ), we have written a Data Storage Interface (DSI) for read/write access to HPSS. This DSI would allow an existing application that uses a GridFTP compliant client to utilize an HPSS data resources. This DSI is currently in testing. Due to changes in the HPSS security mechanisms, it requires HPSS 6.2 or later, which is due to be released in Q4 2005. Distribution for the DSI has not been worked out yet, but it will *probably* be available from both Globus and the HPSS collaboration. While this code will be open source, it requires underlying HPSS libraries which are NOT open source (proprietary).
| ||Note |
|This is a purely server side change, the client does not know what DSI is running, so only a site that is already running HPSS and wants to allow GridFTP access needs to worry about access to these proprietary libraries. |
2.2.3. Accessing data in SRB The following information is helpful if you want to use GridFTP to access data in SRB. Architecturally, the Globus GridFTP server can be divided into 3 modules:
In the GT4.0.x implementation, the data transform module and the DSI have been merged, although we plan to have separate, chainable, data transform modules in the future.
- the GridFTP protocol module,
- the (optional) data transform module, and
- the Data Storage Interface (DSI).
| ||Note |
|This architecture does NOT apply to the WU-FTPD implementation (GT3.2.1 and lower). |
18.104.22.168. GridFTP Protocol Module The GridFTP protocol module is the module that reads and writes to the network and implements the GridFTP protocol. This module should not need to be modified since to do so would make the server non-protocol compliant, and unable to communicate with other servers.
22.214.171.124. Data Transform Functionality The data transform functionality is invoked by using the ERET (extended retrieve) and ESTO (extended store) commands. It is seldom used and bears careful consideration before it is implemented, but in the right circumstances can be very useful. In theory, any computation could be invoked this way, but it was primarily intended for cases where some simple pre-processing (such as a partial get or sub-sampling) can greatly reduce the network load. The disadvantage to this is that you remove any real option for planning, brokering, etc., and any significant computation could adversely affect the data transfer performance. Note that the client must also support the ESTO/ERET functionality as well.
126.96.36.199. Data Storage Interface (DSI) / Data Transform module The Data Storage Interface (DSI) / Data Transform module knows how to read and write to the "local" storage system and can optionally transform the data. We put local in quotes because in a complicated storage system, the storage may not be directly attached, but for performance reasons, it should be relatively close (for instance on the same LAN). The interface consists of functions to be implemented such as send (get), receive (put), command (simple commands that simply succeed or fail like mkdir), etc.. Once these functions have been implemented for a specific storage system, a client should not need to know or care what is actually providing the data. The server can either be configured specifically with a specific DSI, i.e., it knows how to interact with a single class of storage system, or one particularly useful function for the ESTO/ERET functionality mentioned above is to load and configure a DSI on the fly.
188.8.131.52. SRB info Last Update: August 2005 Working with the SRB team at the San Diego Supercomputing Center, we have written a Data Storage Interface (DSI) for read/write access to data in the Storage Resource Broker (SRB) (http://www.npaci.edu/DICE/SRB). This DSI will enable GridFTP compliant clients to read and write data to an SRB server, similar in functionality to the sput/sget commands. This DSI is currently in testing and is not yet publicly available, but will be available from both the SRB web site (here) and the Globus web site (here). It will also be included in the next stable release of the toolkit. We are working on performance tests, but early results indicate that for wide area network (WAN) transfers, the performance is comparable. When might you want to use this functionality:
- You have existing tools that use GridFTP clients and you want to access data that is in SRB
- You have distributed data sets that have some of the data in SRB and some of the data available from GridFTP servers.
2.2.4. Accessing data in some other non-POSIX data source The following information is helpful If you want to use GridFTP to access data in a non-POSIX data source. Architecturally, the Globus GridFTP server can be divided into 3 modules:
In the GT4.0.x implementation, the data transform module and the DSI have been merged, although we plan to have separate, chainable, data transform modules in the future.
- the GridFTP protocol module,
- the (optional) data transform module, and
- the Data Storage Interface (DSI).
| ||Note |
|This architecture does NOT apply to the WU-FTPD implementation (GT3.2.1 and lower). |
184.108.40.206. GridFTP Protocol Module The GridFTP protocol module is the module that reads and writes to the network and implements the GridFTP protocol. This module should not need to be modified since to do so would make the server non-protocol compliant, and unable to communicate with other servers.
220.127.116.11. Data Transform Functionality The data transform functionality is invoked by using the ERET (extended retrieve) and ESTO (extended store) commands. It is seldom used and bears careful consideration before it is implemented, but in the right circumstances can be very useful. In theory, any computation could be invoked this way, but it was primarily intended for cases where some simple pre-processing (such as a partial get or sub-sampling) can greatly reduce the network load. The disadvantage to this is that you remove any real option for planning, brokering, etc., and any significant computation could adversely affect the data transfer performance. Note that the client must also support the ESTO/ERET functionality as well.
18.104.22.168. Data Storage Interface (DSI) / Data Transform module
The Data Storage Interface (DSI) / Data Transform module knows how to read and write to the "local" storage system and can optionally transform the data. We put local in quotes because in a complicated storage system, the storage may not be directly attached, but for performance reasons, it should be relatively close (for instance on the same LAN).
The interface consists of functions to be implemented such as send (get), receive (put), command (simple commands that simply succeed or fail like mkdir), etc..
Once these functions have been implemented for a specific storage system, a client should not need to know or care what is actually providing the data. The server can either be configured specifically with a specific DSI, i.e., it knows how to interact with a single class of storage system, or one particularly useful function for the ESTO/ERET functionality mentioned above is to load and configure a DSI on the fly. 3. Command line tools
Please see the GridFTP Command Reference .
"... Download the authoritative guide: Cloud Computing 2018: Using the Cloud to Transform Your Business ..."
"... Download the authoritative guide: Cloud Computing 2018: Using the Cloud to Transform Your Business ..."
"... Edge computing is a term you are going to hear more of in the coming years because it precedes another term you will be hearing a lot, the Internet of Things (IoT). You see, the formally adopted definition of edge computing is a form of technology that is necessary to make the IoT work. ..."
"... Tech research firm IDC defines edge computing is a "mesh network of micro data centers that process or store critical data locally and push all received data to a central data center or cloud storage repository, in a footprint of less than 100 square feet." ..."
Download the authoritative guide: Cloud Computing 2018: Using the Cloud to Transform Your Business
The term cloud computing is now as firmly lodged in our technical lexicon as email and Internet, and the concept has taken firm hold in business as well. By 2020, Gartner estimates that a "no cloud" policy will be as prevalent in business as a "no Internet" policy. Which is to say no one who wants to stay in business will be without one.
You are likely hearing a new term now, edge computing . One of the problems with technology is terms tend to come before the definition. Technologists (and the press, let's be honest) tend to throw a word around before it is well-defined, and in that vacuum come a variety of guessed definitions, of varying accuracy.
Edge computing is a term you are going to hear more of in the coming years because it precedes another term you will be hearing a lot, the Internet of Things (IoT). You see, the formally adopted definition of edge computing is a form of technology that is necessary to make the IoT work.
Tech research firm IDC defines edge computing is a "mesh network of micro data centers that process or store critical data locally and push all received data to a central data center or cloud storage repository, in a footprint of less than 100 square feet."
It is typically used in IoT use cases, where edge devices collect data from IoT devices and do the processing there, or send it back to a data center or the cloud for processing. Edge computing takes some of the load off the central data center, reducing or even eliminating the processing work at the central location.IoT Explosion in the Cloud Era
To understand the need for edge computing you must understand the explosive growth in IoT in the coming years, and it is coming on big. There have been a number of estimates of the growth in devices, and while they all vary, they are all in the billions of devices.
- Gartner estimates there were 6.4 billion connected devices in 2016 will it reach 20.8 billion by 2020. It estimates that in 2016, 5.5 million new "things" were connected every day.
- IDC predicts global IoT revenue will grow from $2.71 billion in 2015 to $7.065 billion by 2020, with the total installed base of devices reaching 28.1 billion in 2020.
- IHS Markit forecasts that the IoT market will grow from an installed base of 15.4 billion devices in 2015 to 30.7 billion devices by 2020 and 75.4 billion in 2025.
- McKinsey estimates the total IoT market size was about $900 million in 2015 and will grow to $3.7 billion by 2020.
This is taking place in a number of areas, most notably cars and industrial equipment. Cars are becoming increasingly more computerized and more intelligent. Gone are the days when the "Check engine" warning light came on and you had to guess what was wrong. Now it tells you which component is failing.
The industrial sector is a broad one and includes sensors, RFID, industrial robotics, 3D printing, condition monitoring, smart meters, guidance, and more. This sector is sometimes called the Industrial Internet of Things (IIoT) and the overall market is expected to grow from $93.9 billion in 2014 to $151.01 billion by 2020.
All of these sensors are taking in data but they are not processing it. Your car does some of the processing of sensor data but much of it has to be sent in to a data center for computation, monitoring and logging.
The problem is that this would overload networks and data centers. Imaging the millions of cars on the road sending in data to data centers around the country. The 4G network would be overwhelmed, as would the data centers. And if you are in California and the car maker's data center is in Texas, that's a long round trip.
"... Congestion problems lead to miserable performance. We have one WAN pipe, typically 1.5 Mbps to 10 MBps ..."
However, cloud-hosted information assets must still be accessed by users over existing WAN infrastructures, where there are performance issues due to bandwidth and latency constraints.
THE EXTREMELY UNFUNNY PART - UP TO 20x SLOWER
- The "distance" problem
- "Skinny Straw" problem
- Latency and application chattiness
- Slower application access
Thousands of companies Millions of users
♦ Per-unit provisioning costs does not decrease much with size after, say, 100 units.
> Cloud data centers are potentially "far away"
♦ Cloud infrastructure supports many enterprises
♦ Large scale drives lower per-unit cost for data center
> All employees will be "remote" from their data
♦ Even single-location companies will be remote from their data
♦ HQ employees previously local to servers, but not with Cloud model
> Lots of data needs to be sent over limited WAN bandwidth
Congestion problems lead to miserable performance. We have one WAN pipe, typically 1.5 Mbps to 10 MBps
- Web Apps
- Database U J
> Disk-based deduplication technology
♦ Identify redundant data at the byte level, not application (e.g., file) level
♦ Use disks to store vast dictionaries of byte sequences for long periods of time
♦ Use symbols to transfer repetitive sequences of byte-level raw data
♦ Only deduplicated data stored on disk
Troubleshooting MPLS Networks
How should you troubleshoot WAN performance issues. Your MPLS or VPLS network and your clients in field offices are complaining about slow WAN performance. Your network should be performing better and you can't figure out what the problem is. You can contact SD-WAN-Experts to have their engineers solve your problem, but you want to try to solve the problems yourself.
- The first thing to check, seems trivial, but you need to confirm that the ports on your router and switch ports are configured for the same speed and duplex. Log into your switches and check the logs for mismatches of speed or duplex. Auto-negotiation sometimes does not work properly, so a 10M port connected to a 100M port is mismatched. Or you might have a half-duplex port connected to a full-duplex port. Don't assume that a 10/100/1000 port is auto-negotiating correctly!
- Is your WAN performance problem consistent? Does it occur at roughly the same time of day? Or is it completely random? If you don't have the monitoring tools to measure this, you are at a big disadvantage in resolving the issues on your own.
- Do you have Class of Service configured on your WAN? Do you have DSCP configured on your LAN? What is the mapping of your DSCP values to CoS?
- What kind of applications are traversing your WAN? Are there specific apps that work better than others?
- Have your reviewed bandwidth utilization on your carrier's web portal to determine if you are saturating the MPLS port of any locations? Even brief peaks will be enough to generate complaints. Large files, such as CAD drawings, can completely saturate a WAN link.
- Are you backing up or synchronizing data over the WAN? Have you confirmed 100% that this work is completed before the work day begins.
- Might your routing be taking multiple paths and not the most direct path? Look at your routing tables.
- Next, you want to see long term trend statistics. This means monitoring the SNMP streams from all your routers, using tools such as MRTG, NTOP or Cacti. A two week sampling should provide a very good picture of what is happening on your network to help troubleshoot your WAN.
NTOP allows you to
- Sort network traffic according to many protocols
- Show network traffic sorted according to various criteria
- Display traffic statistics
- Store on disk persistent traffic statistics in RRD format
- Identify the identity (e.g. email address) of computer users
- Passively (i.e. without sending probe packets) identify the host OS
- Show IP traffic distribution among the various protocols
- Analyse IP traffic and sort it according to the source/destination
- Display IP Traffic Subnet matrix (who's talking to who?)
- Report IP protocol usage sorted by protocol type
- Act as a NetFlow/ sFlow collector for flows generated by routers (e.g. Cisco and Juniper) or switches (e.g. Foundry Networks)
- Produce RMON-like network traffic statistic
MRTG (Multi-Router Traffic Grapher) provides easy to understand graphs of your network bandwidth utilization.
Cacti requires a MySQL database. It is a complete network graphing solution designed to harness the power of RRDTool 's data storage and graphing functionality. Cacti provides a fast poller, advanced graph templating, multiple data acquisition methods, and user management features out of the box. All of this is wrapped in an intuitive, easy to use interface that makes sense for LAN-sized installations up to complex networks with hundreds of devices.
Both NTOP and MRTG are freeware applications to help troubleshoot your WAN that will run on the freeware versions of Linux. As a result, they can be installed on almost any desktop computer that has out-lived its value as a Windows desktop machine. If you are skilled with Linux and networking, and you have the time, you can install this monitoring system on your own. You will need to get your carrier to provide read-only access to your router SNMP traffic.
But you might find it more cost effective to have the engineers at SD-WAN-Experts do the work for you. All you need to do is provide an available machine with a Linux install (Ubuntu, CentOS, RedHat, etc) with remote access via a VPN. Our engineers will then download all the software remotely, install and configure the machine. When we are done with the monitoring, beside understanding how to solve your problem (and solving it!) you will have your own network monitoring system installed for your use on a daily basis. We'll teach you how to use it, which is quite simple using the web based tools, so you can view it from any machine on your network.
If you need assistance in troubleshooting your wide area network, contact SD-WAN-Experts today !
You might also find these troubleshooting tips of interest;
Troubleshooting MPLS Network Performance Issues
Packet Loss and How It Affects Performance
Troubleshooting VPLS and Ethernet Tunnels over MPLS
Storage in private clouds
Storage is one of the most popular uses of cloud computing, particularly for consumers. The user-friendly design of service-based companies have helped make "cloud" a pretty normal term -- even reaching meme status in 2016.
However, cloud storage means something very different to businesses. Big data and the Internet of Things (IoT) have made it difficult to appraise the value of data until long after it's originally stored -- when finding that piece of data becomes the key to revealing valuable business insights or unlocking an application's new feature. Even after enterprises decide where to store their data in the cloud (on-premise, off-premise, public, or private), they still have to decide how they're going to store it. What good is data that can't be found?
It's common to store data in the cloud using software-defined storage . Software-defined storage decouples storage software from hardware so you can abstract and consolidate storage capacity in a cloud. It allows you to scale beyond whatever individual hardware components your cloud is built on.
Two of the more common software-defined storage solutions include Ceph for structured data and Gluster for unstructured data. Ceph is a massively scalable, programmable storage system that works well with clouds -- particularly those deployed using OpenStack ® -- because of its ability to unify object, block, and file storage into 1 pool of resources. Gluster is designed to handle the requirements of traditional file storage and is particularly adept at provisioning and managing elastic storage for container-based applications.
"... The recent widespread of edge computing in some 5G showcases, like the major sports events, has generated the ongoing discussion about the possibility of edge computing to replace cloud computing. ..."
"... For instance, Satya Nadella, the CEO of Microsoft, announced in Microsoft Build 2017 that the company will focus its strategy on edge computing. Indeed, edge computing will be the key for the success of smart home and driverless vehicles ..."
"... the edge will be the first to process and store the data generated by user devices. This will reduce the latency for the data to travel to the cloud. In other words, the edge optimizes the efficiency for the cloud. ..."
The recent widespread of edge computing in some 5G showcases, like the major sports events, has generated the ongoing discussion about the possibility of edge computing to replace cloud computing.
In fact, there have been announcements from global tech leaders like Nokia and Huawei demonstrating increased efforts and resources in developing edge computing.
For instance, Satya Nadella, the CEO of Microsoft, announced in Microsoft Build 2017 that the company will focus its strategy on edge computing. Indeed, edge computing will be the key for the success of smart home and driverless vehicles.
... ... ...
Cloud or edge, which will lead the future?
The answer to that question is "Cloud – Edge Mixing". The cloud and the edge will complement each other to offer the real IoT experience. For instance, while the cloud coordinates all the technology and offers SaaS to users, the edge will be the first to process and store the data generated by user devices. This will reduce the latency for the data to travel to the cloud. In other words, the edge optimizes the efficiency for the cloud.
It is strongly suggested to implement open architecture white-box servers for both cloud and edge, to minimize the latency for cloud-edge synchronization and optimize the compatibility between the two. For example, Lanner Electronics offers a wide range of Intel x86 white box appliances for data centers and edge uCPE/vCPE.
The hybrid cloud
A hybrid cloud is a combination of a private cloud combined with the use of public cloud services where one or several touch points exist between the environments. The goal is to combine services and data from a variety of cloud models to create a unified, automated, and well-managed computing environment.
Combining public services with private clouds and the data center as a hybrid is the new definition of corporate computing. Not all companies that use some public and some private cloud services have a hybrid cloud. Rather, a hybrid cloud is an environment where the private and public services are used together to create value.
A cloud is hybrid
If a company uses a public development platform that sends data to a private cloud or a data center–based application.
When a company leverages a number of SaaS (Software as a Service) applications and moves data between private or data center resources.
When a business process is designed as a service so that it can connect with environments as though they were a single environment.
A cloud is not hybrid
If a few developers in a company use a public cloud service to prototype a new application that is completely disconnected from the private cloud or the data center.
If a company is using a SaaS application for a project but there is no movement of data from that application into the company's data center.
"... "There's a big transformation happening," says Thomas Humphrey, segment director for edge computing at APC . "Technologies like IoT have started to require that some local computing and storage happen out in that distributed IT architecture." ..."
"... In retail, for example, edge computing will become more important as stores find success with IoT technologies such as mobile beacons, interactive mirrors and real-time tools for customer experience, behavior monitoring and marketing . ..."
Enterprises are deploying self-contained micro data centers to power computing at the network edge.
The location for data processing has changed significantly throughout the history of computing. During the mainframe era, data was processed centrally, but client/server architectures later decentralized computing. In recent years, cloud computing centralized many processing workloads, but digital transformation and the Internet of Things are poised to move computing to new places, such as the network edge .
"There's a big transformation happening," says Thomas Humphrey, segment director for edge computing at APC . "Technologies like IoT have started to require that some local computing and storage happen out in that distributed IT architecture."
For example, some IoT systems require processing of data at remote locations rather than a centralized data center , such as at a retail store instead of a corporate headquarters.
To meet regulatory requirements and business needs, IoT solutions often need low latency, high bandwidth, robust security and superior reliability . To meet these demands, many organizations are deploying micro data centers: self-contained solutions that provide not only essential infrastructure, but also physical security, power and cooling and remote management capabilities.
"Digital transformation happens at the network edge, and edge computing will happen inside micro data centers ," says Bruce A. Taylor, executive vice president at Datacenter Dynamics . "This will probably be one of the fastest growing segments -- if not the fastest growing segment -- in data centers for the foreseeable future."What Is a Micro Data Center?
Delivering the IT capabilities needed for edge computing represents a significant challenge for many organizations, which need manageable and secure solutions that can be deployed easily, consistently and close to the source of computing . Vendors such as APC have begun to create comprehensive solutions that provide these necessary capabilities in a single, standardized package.
"From our perspective at APC, the micro data center was a response to what was happening in the market," says Humphrey. "We were seeing that enterprises needed more robust solutions at the edge."
Most micro data center solutions rely on hyperconverged infrastructure to integrate computing, networking and storage technologies within a compact footprint . A typical micro data center also incorporates physical infrastructure (including racks), fire suppression, power, cooling and remote management capabilities. In effect, the micro data center represents a sweet spot between traditional IT closets and larger modular data centers -- giving organizations the ability to deploy professional, powerful IT resources practically anywhere .Standardized Deployments Across the Country
Having robust IT resources at the network edge helps to improve reliability and reduce latency, both of which are becoming more and more important as analytics programs require that data from IoT deployments be processed in real time .
"There's always been edge computing," says Taylor. "What's new is the need to process hundreds of thousands of data points for analytics at once."
Standardization, redundant deployment and remote management are also attractive features, especially for large organizations that may need to deploy tens, hundreds or even thousands of micro data centers. "We spoke to customers who said, 'I've got to roll out and install 3,500 of these around the country,'" says Humphrey. "And many of these companies don't have IT staff at all of these sites." To address this scenario, APC designed standardized, plug-and-play micro data centers that can be rolled out seamlessly. Additionally, remote management capabilities allow central IT departments to monitor and troubleshoot the edge infrastructure without costly and time-intensive site visits.
In part because micro data centers operate in far-flung environments, security is of paramount concern. The self-contained nature of micro data centers ensures that only authorized personnel will have access to infrastructure equipment , and security tools such as video surveillance provide organizations with forensic evidence in the event that someone attempts to infiltrate the infrastructure.How Micro Data Centers Can Help in Retail, Healthcare
Micro data centers make business sense for any organization that needs secure IT infrastructure at the network edge. But the solution is particularly appealing to organizations in fields such as retail, healthcare and finance , where IT environments are widely distributed and processing speeds are often a priority.
In retail, for example, edge computing will become more important as stores find success with IoT technologies such as mobile beacons, interactive mirrors and real-time tools for customer experience, behavior monitoring and marketing .
"It will be leading-edge companies driving micro data center adoption, but that doesn't necessarily mean they'll be technology companies," says Taylor. "A micro data center can power real-time analytics for inventory control and dynamic pricing in a supermarket."
In healthcare, digital transformation is beginning to touch processes and systems ranging from medication carts to patient records, and data often needs to be available locally; for example, in case of a data center outage during surgery. In finance, the real-time transmission of data can have immediate and significant financial consequences. And in both of these fields, regulations governing data privacy make the monitoring and security features of micro data centers even more important.
Micro data centers also have enormous potential to power smart city initiatives and to give energy companies a cost-effective way of deploying resources in remote locations , among other use cases.
"The proliferation of edge computing will be greater than anything we've seen in the past," Taylor says. "I almost can't think of a field where this won't matter."
Learn more about how solutions and services from CDW and APC can help your organization overcome its data center challenges.Micro Data Centers Versus IT Closets
Think the micro data center is just a glorified update on the traditional IT closet? Think again.
"There are demonstrable differences," says Bruce A. Taylor, executive vice president at Datacenter Dynamics. "With micro data centers, there's a tremendous amount of computing capacity in a very small, contained space, and we just didn't have that capability previously ."
APC identifies three key differences between IT closets and micro data centers:
- Difference #1: Uptime Expectations. APC notes that, of the nearly 3 million IT closets in the U.S., over 70 percent report outages directly related to human error. In an unprotected IT closet, problems can result from something as preventable as cleaning staff unwittingly disconnecting a cable. Micro data centers, by contrast, utilize remote monitoring, video surveillance and sensors to reduce downtime related to human error.
- Difference #2: Cooling Configurations. The cooling of IT wiring closets is often approached both reactively and haphazardly, resulting in premature equipment failure. Micro data centers are specifically designed to assure cooling compatibility with anticipated loads.
- Difference #3: Power Infrastructure. Unlike many IT closets, micro data centers incorporate uninterruptible power supplies, ensuring that infrastructure equipment has the power it needs to help avoid downtime.
Calvin Hennick is a freelance journalist who specializes in business and technology writing. He is a contributor to the CDW family of technology magazines.
"... most of the Office365 deployments face network related problems - typically manifesting as screen freezes. Limited WAN optimization capability further complicates the problems for most SaaS applications. ..."
"... Why enterprises overlook the importance of strategically placing cloud gateways ..."
About this webinar Major research highlights that most of the Office365 deployments face network related problems - typically manifesting as screen freezes. Limited WAN optimization capability further complicates the problems for most SaaS applications. To compound the issue, different SaaS applications issue different guidelines for solving performance issues. We will investigate the major reasons for these problems.
SD-WAN provides an essential set of features that solves these networking issues related to Office 365 and SaaS applications. This session will cover the following major topics:
- How traditional networks impair O365 performance, and why?
- Best practice architectures for different kinds of SaaS applications
"... We already know that computing at the edge pushes most of the data processing out to the edge of the network, close to the source of the data. Then it's a matter of dividing the processing between the edge and the centralized system, meaning a public cloud such as Amazon Web Services, Google Cloud, or Microsoft Azure. ..."
"... The goal is to process near the device the data that it needs quickly, such as to act on. There are hundreds of use cases where reaction time is the key value of the IoT system, and consistently sending the data back to a centralized cloud prevents that value from happening. ..."
The internet of things is real, and it's a real part of the cloud. A key challenge is how you can get data processed from so many devices. Cisco Systems predicts that cloud traffic is likely to rise nearly fourfold by 2020, increasing 3.9 zettabytes (ZB) per year in 2015 (the latest full year for which data is available) to 14.1ZB per year by 2020.
As a result, we could have the cloud computing perfect storm from the growth of IoT. After all, IoT is about processing device-generated data that is meaningful, and cloud computing is about using data from centralized computing and storage. Growth rates of both can easily become unmanageable.
So what do we do? The answer is something called "edge computing." We already know that computing at the edge pushes most of the data processing out to the edge of the network, close to the source of the data. Then it's a matter of dividing the processing between the edge and the centralized system, meaning a public cloud such as Amazon Web Services, Google Cloud, or Microsoft Azure.
That may sound a like a client/server architecture, which also involved figuring out what to do at the client versus at the server. For IoT and any highly distributed applications, you've essentially got a client/network edge/server architecture going on, or -- if your devices can't do any processing themselves, a network edge/server architecture.
The goal is to process near the device the data that it needs quickly, such as to act on. There are hundreds of use cases where reaction time is the key value of the IoT system, and consistently sending the data back to a centralized cloud prevents that value from happening.
You would still use the cloud for processing that is either not as time-sensitive or is not needed by the device, such as for big data analytics on data from all your devices.
There's another dimension to this: edge computing and cloud computing are two very different things. One does not replace the other. But too many articles confuse IT pros by suggesting that edge computing will displace cloud computing. It's no more true than saying PCs would displace the datacenter.
It makes perfect sense to create purpose-built edge computing-based applications, such as an app that places data processing in a sensor to quickly process reactions to alarms. But you're not going to place your inventory-control data and applications at the edge -- moving all compute to the edge would result in a distributed, unsecured, and unmanageable mess.
All the public cloud providers have IoT strategies and technology stacks that include, or will include, edge computing. Edge and cloud computing can and do work well together, but edge computing is for purpose-built systems with special needs. Cloud computing is a more general-purpose platform that also can work with purpose-built systems in that old client/server model.
David S. Linthicum is a chief cloud strategy officer at Deloitte Consulting, and an internationally recognized industry expert and thought leader. His views are his own.
"... GridFTP is a high-performance, secure, reliable data transfer protocol optimized for high-bandwidth wide-area networks ..."
The open source Globus® Toolkit is a fundamental enabling technology for the "Grid," letting people share computing power, databases, and other tools securely online across corporate, institutional, and geographic boundaries without sacrificing local autonomy. The toolkit includes software services and libraries for resource monitoring, discovery, and management, plus security and file management. In addition to being a central part of science and engineering projects that total nearly a half-billion dollars internationally, the Globus Toolkit is a substrate on which leading IT companies are building significant commercial Grid products.
The toolkit includes software for security, information infrastructure, resource management, data management, communication, fault detection, and portability. It is packaged as a set of components that can be used either independently or together to develop applications. Every organization has unique modes of operation, and collaboration between multiple organizations is hindered by incompatibility of resources such as data archives, computers, and networks. The Globus Toolkit was conceived to remove obstacles that prevent seamless collaboration. Its core services, interfaces and protocols allow users to access remote resources as if they were located within their own machine room while simultaneously preserving local control over who can use resources and when.
The Globus Toolkit has grown through an open-source strategy similar to the Linux operating system's, and distinct from proprietary attempts at resource-sharing software. This encourages broader, more rapid adoption and leads to greater technical innovation, as the open-source community provides continual enhancements to the product.
Essential background is contained in the papers " Anatomy of the Grid " by Foster, Kesselman and Tuecke and " Physiology of the Grid " by Foster, Kesselman, Nick and Tuecke.Acclaim for the Globus Toolkit
From version 1.0 in 1998 to the 2.0 release in 2002 and now the latest 4.0 version based on new open-standard Grid services, the Globus Toolkit has evolved rapidly into what The New York Times called "the de facto standard" for Grid computing. In 2002 the project earned a prestigious R&D 100 award, given by R&D Magazine in a ceremony where the Globus Toolkit was named "Most Promising New Technology" among the year's top 100 innovations. Other honors include project leaders Ian Foster of Argonne National Laboratory and the University of Chicago, Carl Kesselman of the University of Southern California's Information Sciences Institute (ISI), and Steve Tuecke of Argonne being named among 2003's top ten innovators by InfoWorld magazine, and a similar honor from MIT Technology Review, which named Globus Toolkit-based Grid computing one of "Ten Technologies That Will Change the World." The Globus Toolkit also GridFTP is a high-performance, secure, reliable data transfer protocol optimized for high-bandwidth wide-area networks . The GridFTP protocol is based on FTP, the highly-popular Internet file transfer protocol. We have selected a set of protocol features and extensions defined already in IETF RFCs and added a few additional features to meet requirements from current data grid projects.
The following guides are available for this component:
Requires See Also Provides
Required By Download Install Howto
|Name ||Value |
|globus-gridftp-server-control ||= 6.1-1.el7 |
|globus-gridftp-server-control(x86-64) ||= 6.1-1.el7 |
|libglobus_gridftp_server_control.so.0()(64bit) ||- |
- Download the latest epel-release rpm from
- Install epel-release rpm:
# rpm -Uvh epel-release*rpm
- Install globus-gridftp-server-control rpm package:
# yum install globus-gridftp-server-control
2018-04-07 - Mattias Ellert <firstname.lastname@example.org> - 6.1-1
- GT6 update: Don't error if acquire_cred fails when vhost env is set
Aspera Sync is an elite, versatile, multi-directional no concurrent record replication and synchronization. It is intended to conquer the execution and versatility inadequacies of conventional synchronization instruments like Rsync. Aspera Sync can scale up and out for most extreme rate replication and synchronization over WANs. Prominent capacities are The FASP advantage, superior, smart trade for Rsync, underpins complex synchronization arrangements, propelled record taking care of, and so on. Aspera Sync is reason worked by Aspera for elite, versatile, multi-directional offbeat record replication and synchronization. Intended to beat the execution and adaptability deficiencies of conventional synchronization instruments like Rsync, Aspera Sync can scale up and out for greatest pace replication and synchronization over WANs, for now,'s biggest vast information record stores -- from a great many individual documents to the most significant document sizes. Hearty reinforcement and recuperation strategies secure business necessary information and frameworks so undertakings can rapidly recoup necessary documents, structures or a whole site in the occasion if a calamity. Be that as it may, these strategies can be undermined by average exchange speeds amongst essential and reinforcement locales, bringing about fragmented reinforcements and augmented recuperation times. With FASP – controlled transactions, replication fits inside the little operational window so you can meet your recuperation point objective (RPO) and recovery time objective (RTO).1. Syncthing Syncthing replaces exclusive synchronize and cloud administrations with something open, reliable and decentralized. Your information is your information alone, and you should pick where it is put away if it is imparted to some outsider and how it's transmitted over the Internet. Syncthing is a record sharing application that permits you to share reports between various gadgets in an advantageous way. Its online Graphical User Interface (GUI) makes it conceivable Website Syncthing Alternatives
"... These C level guys see cloud services � applications, data, backup, service desk � as a great way to free up a blockage in how IT service is being delivered on premise. ..."
"... IMHO there is a big difference between management of IT and management of IT service. Rarely do you get people who can do both. ..."
...Cloud introduces a whole new ball game and will no doubt perpetuate Putts Law for ever more. Why?
Well unless 100% of IT infrastructure goes up into the clouds ( unlikely for any organization with a history ; likely for a new organization ( probably micro small ) that starts up in the next few years ) the 'art of IT management' will demand even more focus and understanding.
I always think a great acid test of Putts Law is to look at one of the two aspects of IT management
- Show me a simple process that you follow each day that delivers an aspect of IT service i.e. how to buy a piece of IT stuff, or a way to report a fault
- Show me how you manage a single entity on the network i.e. a file server, a PC, a network switch
Usually the answers ( which will be different from people on the same team, in the same room and from the same person on different days !) will give you an insight to Putts Law.
Childs play for most of course who are challenged with some real complex management situations such as data center virtualization projects, storage explosion control, edge device management, backend application upgrades, global messaging migrations and B2C identity integration. But of course if its evidenced that they seem to be managing (simple things ) without true understanding one could argue 'how the hell can they be expected to manage what they understand with the complex things?' Fair point?
Of course many C level people have an answer to Putts Law. Move the problem to people who do understand what they manage. Professionals who provide cloud versions of what the C level person struggles to get a professional service from. These C level guys see cloud services � applications, data, backup, service desk � as a great way to free up a blockage in how IT service is being delivered on premise. And they are right ( and wrong ).
... ... ...
( Quote attributed to Archibald Putt author of Putt's Law and the Successful Technocrat: How to Win in the Information Age )
rowan says: March 9, 2012 at 9:03 am
stephen777 says: March 31, 2012 at 7:32 am
IMHO there is a big difference between management of IT and management of IT service. Rarely do you get people who can do both. Understanding inventory, disk space, security etc is one thing; but understanding the performance of apps and user impact is another ball game. Putts Law is alive and well in my organisation. TGIF.
Rowan in Belfast.
Rowan is right I used to be an IT Manager but now my title is Service Delivery Manager. Why? Because we had a new CTO who changed how people saw what we did. I ve been doing this new role for 5 years and I really do understand what i don't manage. LOL
ControlledExperiments Ars Centurion reply 5 hours ago I am not your friend wrote: I just can't comprehend that price. Cloud has a rich future, but I didn't even know Red Hat had any presence there, let alone $35 billion worth.
Yeah the price is nuts, reminiscent of FB's WhatsApp purchase, and maybe their Instagram purchase.
If you actually look at the revenue Amazon gets from AWS or Microsoft from Azure, it's not that much, relatively speaking. For Microsoft, it's nothing compared to Windows and Office revenue, and I'm not sure where the growth is supposed to come from. It seems like most everyone who wants to be on the cloud is already there, and vulns like Spectre and Meltdown broke the sacred VM wall, so...
realist , Oct 27, 2018 10:27:04 AM | link
The first wave of the computer revolution created stand-alone systems. The current wave is their combination into new and much larger ones.
Posted by b on October 26, 2018 at 02:18 PM
Not strictly correct. It has come full circle in sense. We started with standalone machines; then we had central mainframes with dumb terminals; then
the first networks of these centralized servers (XeroxParc's efforts being more forward looking and an exception); then PCs; then internet; and now (per design and not by merit, imo)
We are effectively back to the centralized systems ("clouds" controlled by Amazon (CIA), Google (NSA)). The peripheral components of autonomous networked systems are akin to sensory devices with heavy computational lifting in the (local) regime controlled servers.
By this, I mean moral codes no longer have any objective basis but are evermore easily re-wired by the .002 to suit their purposes
Posted by: donkeytale | Oct 27, 2018 6:32:11 AM | 69
I question the implied "helpless to resist reprogramming" notion of your statement. You mentioned God. Please note that God has endowed each and every one of us with a moral compass. You mentioned Jesus. Didn't he say something about "you generation of vipers"? I suggest you consider that ours is a morally degenerate generation and is getting precisely what it wants and deserves.
Posted by msmash on Thursday September 07, 2017
Numerous Slashdot readers are reporting that they are facing issues access Google Drive, the productivity suite from the Mountain View-based company. Google's dashboard confirms that Drive is facing outage .
Third-party web monitoring tool DownDetector also reports thousands of similar complaints from users. The company said, "Google Drive service has already been restored for some users, and we expect a resolution for all users in the near future.
Please note this time frame is an estimate and may change. Google Drive is not loading files and results in a failures for a subset of users."
The intelligence community is about to get the equivalent of an adrenaline shot to the chest. This summer, a $600 million computing cloud developed by Amazon Web Services for the Central Intelligence Agency over the past year will begin servicing all 17 agencies that make up the intelligence community. If the technology plays out as officials envision, it will usher in a new era of cooperation and coordination, allowing agencies to share information and services much more easily and avoid the kind of intelligence gaps that preceded the Sept. 11, 2001, terrorist attacks.
For the first time, agencies within the intelligence community will be able to order a variety of on-demand computing and analytic services from the CIA and National Security Agency. What's more, they'll only pay for what they use.
The vision was first outlined in the Intelligence Community Information Technology Enterprise plan championed by Director of National Intelligence James Clapper and IC Chief Information Officer Al Tarasiuk almost three years ago. Cloud computing is one of the core components of the strategy to help the IC discover, access and share critical information in an era of seemingly infinite data.
For the risk-averse intelligence community, the decision to go with a commercial cloud vendor is a radical departure from business as usual.
In 2011, while private companies were consolidating data centers in favor of the cloud and some civilian agencies began flirting with cloud variants like email as a service, a sometimes contentious debate among the intelligence community's leadership took place.
... ... ... The government was spending more money on information technology within the IC than ever before. IT spending reached $8 billion in 2013, according to budget documents leaked by former NSA contractor Edward Snowden. The CIA and other agencies feasibly could have spent billions of dollars standing up their own cloud infrastructure without raising many eyebrows in Congress, but the decision to purchase a single commercial solution came down primarily to two factors.
"What we were really looking at was time to mission and innovation," the former intelligence official said. "The goal was, 'Can we act like a large enterprise in the corporate world and buy the thing that we don't have, can we catch up to the commercial cycle? Anybody can build a data center, but could we purchase something more?
"We decided we needed to buy innovation," the former intelligence official said. A Groundbreaking Deal
... ... ... The Amazon-built cloud will operate behind the IC's firewall, or more simply: It's a public cloud built on private premises.
Intelligence agencies will be able to host applications or order a variety of on-demand services like storage, computing and analytics. True to the National Institute of Standards and Technology definition of cloud computing, the IC cloud scales up or down to meet the need.
In that regard, customers will pay only for services they actually use, which is expected to generate massive savings for the IC.
"We see this as a tremendous opportunity to sharpen our focus and to be very efficient," Wolfe told an audience at AWS' annual nonprofit and government symposium in Washington. "We hope to get speed and scale out of the cloud, and a tremendous amount of efficiency in terms of folks traditionally using IT now using it in a cost-recovery way."
... ... ... For several years there hasn't been even a close challenger to AWS. Gartner's 2014 quadrant shows that AWS captures 83 percent of the cloud computing infrastructure market.
In the combined cloud markets for infrastructure and platform services, hybrid and private clouds-worth a collective $131 billion at the end of 2013-Amazon's revenue grew 67 percent in the first quarter of 2014, according to Gartner.
While the public sector hasn't been as quick to capitalize on cloud computing as the private sector, government spending on cloud technologies is beginning to jump.
Researchers at IDC estimate federal private cloud spending will reach $1.7 billion in 2014, and $7.7 billion by 2017. In other industries, software services are considered the leading cloud technology, but in the government that honor goes to infrastructure services, which IDC expects to reach $5.4 billion in 2017.
In addition to its $600 million deal with the CIA, Amazon Web Services also does business with NASA, the Food and Drug Administration and the Centers for Disease Control and Prevention. Most recently, the Obama Administration tapped AWS to host portions of HealthCare.gov.
Amazon's S3 web-based storage service is experiencing widespread issues, leading to service that's either partially or fully broken on websites, apps and devices upon which it relies. The AWS offering provides hosting for images for a lot of sites, and also hosts entire websites, and app backends including Nest.
The S3 outage is due to "high error rates with S3 in US-EAST-1," according to Amazon's AWS service health dashboard
, which is where the company also says it's working on "remediating the issue," without initially revealing any further details.
Affected websites and services include Quora, newsletter provider Sailthru, Business Insider, Giphy, image hosting at a number of publisher websites, filesharing in Slack, and many more. Connected lightbulbs, thermostats and other IoT hardware is also being impacted, with many unable to control these devices as a result of the outage.
Amazon S3 is used by around 148,213 websites, and 121,761 unique domains, according to data tracked by SimilarTech
, and its popularity as a content host concentrates specifically in the U.S. It's used by 0.8 percent of the top 1 million websites, which is actually quite a bit smaller than CloudFlare, which is used by 6.2 percent of the top 1 million websites globally � and yet it's still having this much of an effect.
Amazingly, even the status indicators on the AWS service status page rely on S3 for storage of its health marker graphics, hence why the site is still showing all services green despite obvious evidence to the contrary. Update (11:40 AM PT): AWS has fixed the issues with its own dashboard at least � it'll now accurately reflect service status as it continues to attempt to fix the problem
- Update (11:57 AM PT): AWS says it believes they new "understand root cause" of the S3 issues, and are "working hard at repairing." It has not shared specifics of that cause.
- Update (12:15 PM PT): Network intelligence software provider ThousandEyes notes that all the packet loss for the ongoing issue appears to be happening in the Ashburn, VA area. Amazon has an AWS data center in Ashburn, whose exact location was revealed in a news story last year due to a fire during its construction.
- Update (12:54 PM PT): AWS says it's seeing "recovery for S3 object retrievals, listing and deletions" which means you're probably seeing avatars and other visuals assets come back in some spots. The company also says it expects further improvements to error rates within the next hour.
- Update (1:20 PM PT): S3 is now fully recovered in terms of the retrieval, listing and deletion of existing objects, according to the AWS status page, and it's now working on restoring normal operation for the addition of new items to S3-based storage.
- Update (2:10 PM PT): AWS says that it's now fully recovered in terms of resolving the error rates it was seeing, and S3 service is now "operating normally."
Feb 28, 2017 6:03 PM EST
NEW YORK -- Amazon's cloud-computing service, Amazon Web Services, experienced an outage in its eastern U.S. region Tuesday afternoon, causing unprecedented and widespread problems for thousands of websites and apps.
Amazon is the largest provider of cloud computing services in the U.S. Beginning around midday Tuesday on the East Coast, one region of its "S3" service based in Virginia began to experience what Amazon, on its service site, called "increased error rates."
In a statement, Amazon said as of 4 p.m. E.T. it was still experiencing "high error rates" that were "impacting various AWS services."
"We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue," the company said.
But less than an hour later, an update offered good news: "As of 1:49 PM PST, we are fully recovered for operations for adding new objects in S3, which was our last operation showing a high error rate. The Amazon S3 service is operating normally," the company said.
Amazon's Simple Storage Service, or S3, stores files and data for companies on remote servers. It's used for everything from building websites and apps to storing images, customer data and customer transactions.
"Anything you can think about storing in the most cost-effective way possible," is how Rich Mogull, CEO of data security firm Securosis, puts it.
Amazon has a strong track record of stability with its cloud computing service, CNET
senior editor Dan Ackerman told CBS News.
"AWS... is known for having really good 'up time,'" he said, using industry language.
Over time, cloud computing has become a major part of Amazon's empire.
"Very few people host their own web servers anymore, it's all been outsourced to these big providers
, and Amazon is one of the major ones," Ackerman said.
The problem Tuesday affected both "front-end" operations -- meaning the websites and apps that users see -- and back-end data processing that takes place out of sight. Some smaller online services, such as Trello, Scribd and IFTTT, appeared to be down for a while, although all have since recovered.
Some affected websites had fun with the crash, treating it like a snow day:
"... "From a sustainability and availability standpoint, we definitely need to look at our strategy to not be vendor specific, including with Amazon," said Lee. "That's something that we're aware of and are working towards." ..."
"... "Elastic load balances and other services make it easy to set up. However, it's a double-edged sword, because these types of services will also make it harder to be vendor-agnostic. When other cloud platform don't offer the same services, how do you wean yourself off of them?" ..."
"... Multi-year commitments are another trap, he said. And sometimes there's an extra unpleasant twist -- minimum usage requirements that go up in the later years, like balloon payments on a mortgage. ..."
The Amazon outage reminds companies that having all their eggs in one cloud basket might be a risky strategy
"That is the elephant in the room these days," said Lee. "More and more companies are starting to move their services to the cloud providers. I see attackers trying to compromise the cloud provider to get to the information."
If attackers can get into the cloud systems, that's a lot of data they could have access to. But attackers can also go after availability.
"The DDoS attacks are getting larger in scale, and with more IoT systems coming online and being very hackable, a lot of attackers can utilize that as a way to do additional attacks," he said.
And, of course, there's always the possibility of a cloud service outage for other reasons.
The 11-hour outage that Amazon suffered in late February was due to a typo, and affected Netflix, Reddit, Adobe and Imgur, among other sites.
"From a sustainability and availability standpoint, we definitely need to look at our strategy to not be vendor specific, including with Amazon," said Lee. "That's something that we're aware of and are working towards."
The problem is that Amazon offers some very appealing features.
"Amazon has been very good at providing a lot of services that reduce the investment that needs to be made to build the infrastructure," he said. "Elastic load balances and other services make it easy to set up. However, it's a double-edged sword, because these types of services will also make it harder to be vendor-agnostic. When other cloud platform don't offer the same services, how do you wean yourself off of them?"
... ... ... "If you have a containerized approach, you can run in Amazon's container services, or on Azure," said Tim Beerman, CTO at Ensono , a managed services provider that runs its own cloud data center, manages on-premises environments for customers, and also helps clients run in the public cloud.
"That gives you more portability, you can pick something up and move it," he said.
But that, too, requires advance planning.
"It starts with the application," he said. "And you have to write it a certain way."
But the biggest contributing factor to cloud lock-in is data, he said.
"They make it really easy to put the data in, and they're not as friendly about taking that data out," he said.
The lack of friendliness often shows up in the pricing details.
"Usually the price is lower for data transfers coming into a cloud service provider versus the price to move data out," said Thales' Radford.
Multi-year commitments are another trap, he said. And sometimes there's an extra unpleasant twist -- minimum usage requirements that go up in the later years, like balloon payments on a mortgage.
Is your server or servers getting old? Have you pushed it to the end of its lifespan? Have you reached that stage where it's time to do something about it? Join the crowd. You're now at that decision point that so many other business people are finding themselves this year. And the decision is this: do you replace that old server with a new server or do you go to: the cloud.
Everyone's talking about the cloud nowadays so you've got to consider it, right? This could be a great new thing for your company! You've been told that the cloud enables companies like yours to be more flexible and save on their IT costs. It allows free and easy access to data for employees from wherever they are, using whatever devices they want to use. Maybe you've seen the recent survey
by accounting software maker MYOB that found that small businesses that adopt cloud technologies enjoy higher revenues. Or perhaps you've stumbled on this analysis
that said that small businesses are losing money as a result of ineffective IT management that could be much improved by the use of cloud based services. Or the poll
of more than 1,200 small businesses by technology reseller CDW
which discovered that " cloud users cite cost savings, increased efficiency and greater innovation as key benefits" and that " across all industries, storage and conferencing and collaboration are the top cloud services and applications."
So it's time to chuck that old piece of junk and take your company to the cloud, right? Well just hold on.
There's no question that if you're a startup or a very small company or a company that is virtual or whose employees are distributed around the world, a cloud based environment is the way to go. Or maybe you've got high internal IT costs or require more computing power. But maybe that's not you. Maybe your company sells pharmaceutical supplies, provides landscaping services, fixes roofs, ships industrial cleaning agents, manufactures packaging materials or distributes gaskets. You are not featured in Fast Company and you have not been invited to presenting at the next Disrupt conference. But you know you represent the very core of small business in America. I know this too. You are just like one of my company's 600 clients. And what are these companies doing this year when it comes time to replace their servers?
These very smart owners and managers of small and medium sized businesses who have existing applications running on old servers are not going to the cloud. Instead, they've been buying new servers.
Wait, buying new servers? What about the cloud?
At no less than six of my clients in the past 90 days it was time to replace servers. They had all waited as long as possible, conserving cash in a slow economy, hoping to get the most out of their existing machines. Sound familiar? But the servers were showing their age, applications were running slower and now as the companies found themselves growing their infrastructure their old machines were reaching their limit. Things were getting to a breaking point, and all six of my clients decided it was time for a change. So they all moved to cloud, right?
Nope. None of them did. None of them chose the cloud. Why? Because all six of these small business owners and managers came to the same conclusion: it was just too expensive. Sorry media. Sorry tech world. But this is the truth. This is what's happening in the world of established companies.
Consider the options. All of my clients' evaluated cloud based hosting services from Amazon
. They also interviewed a handful of cloud based IT management firms who promised to move their existing applications (Office, accounting, CRM, databases) to their servers and manage them offsite. All of these popular options are viable and make sense, as evidenced by their growth in recent years. But when all the smoke cleared, all of these services came in at about the same price: approximately $100 per month per user. This is what it costs for an existing company to move their existing infrastructure to a cloud based infrastructure in 2013. We've got the proposals and we've done the analysis.
You're going through the same thought process, so now put yourself in their shoes. Suppose you have maybe 20 people in your company who need computer access. Suppose you are satisfied with your existing applications and don't want to go through the agony and enormous expense of migrating to a new cloud based application. Suppose you don't employ a full time IT guy, but have a service contract with a reliable local IT firm.
Now do the numbers: $100 per month x 20 users is $2,000 per month or $24,000 PER YEAR for a cloud based service. How many servers can you buy for that amount? Imagine putting that proposal out to an experienced, battle-hardened, profit generating small business owner who, like all the smart business owners I know, look hard at the return on investment decision before parting with their cash.
For all six of these clients the decision was a no-brainer: they all bought new servers and had their IT guy install them. But can't the cloud bring down their IT costs? All six of these guys use their IT guy for maybe half a day a month to support their servers (sure he could be doing more, but small business owners always try to get away with the minimum). His rate is $150 per hour. That's still way below using a cloud service.
No one could make the numbers work. No one could justify the return on investment. The cloud, at least for established businesses who don't want to change their existing applications, is still just too expensive.
Please know that these companies are, in fact, using some cloud-based applications. They all have virtual private networks setup and their people access their systems over the cloud using remote desktop technologies. Like the respondents in the above surveys, they subscribe to online backup services, share files on DropBox and Microsoft
's file storage, make their calls over Skype, take advantage of Gmail and use collaboration tools like Google
Docs or Box. Many of their employees have iPhones and Droids and like to use mobile apps which rely on cloud data to make them more productive. These applications didn't exist a few years ago and their growth and benefits cannot be denied.
Paul-Henri Ferrand, President of Dell
North America, doesn't see this trend continuing. "Many smaller but growing businesses are looking and/or moving to the cloud," he told me. "There will be some (small businesses) that will continue to buy hardware but I see the trend is clearly toward the cloud. As more business applications become more available for the cloud, the more likely the trend will continue."
"... By Shane Greenstein On Jan 11, 2017 � Add Comment � In Broadband , communication , Esssay , Net Neutrality ..."
"... The bottom line: evenings require far greater capacity than other times of the day. If capacity is not adequate, it can manifest as a bottleneck at many different points in a network-in its backbone, in its interconnection points, or in its last mile nodes. ..."
"... The use of tiers tends to grab attention in public discussion. ISPs segment their users. Higher tiers bring more bandwidth to a household. All else equal, households with higher tiers experience less congestion at peak moments. ..."
"... such firms (typically) find clever ways to pile on fees, and know how to stymie user complaints with a different type of phone tree that makes calls last 45 minutes. Even when users like the quality, the aggressive pricing practices tend to be quite irritating. ..."
"... Some observers have alleged that the biggest ISPs have created congestion issues at interconnection points for purposes of gaining negotiating leverage. These are serious charges, and a certain amount of skepticism is warranted for any broad charge that lacks specifics. ..."
"... Congestion is inevitable in a network with interlocking interests. When one part of the network has congestion, the rest of it catches a cold. ..."
"... More to the point, growth in demand for data should continue to stress network capacity into the foreseeable future. Since not all ISPs will invest aggressively in the presence of congestion, some amount of congestion is inevitable. So, too, is a certain amount of irritation. ..."
Congestion on the Last Mile
By Shane Greenstein
On Jan 11, 2017 � Add Comment
� In Broadband
, Net Neutrality
It has long been recognized that networked services contain weak-link vulnerabilities. That is, the performance of any frontier device depends on the performance of every contributing component and service. This column focuses on one such phenomenon, which goes by the label "congestion." No, this is not a new type of allergy, but, as with a bacteria, many users want to avoid it, especially advanced users of frontier network services.
Congestion arises when network capacity does not provide adequate service during heavy use. Congestion slows down data delivery and erodes application performance, especially for time-sensitive apps such as movies, online videos, and interactive gaming.
Concerns about congestion are pervasive. Embarrassing reports about broadband networks with slow speeds highlight the role of congestion. Regulatory disputes about data caps and pricing tiers question whether these programs limit the use of data in a useful way. Investment analysts focus on the frequency of congestion as a measure of a broadband network's quality.
What economic factors produce congestion? Let's examine the root economic causes.
Congestion arises when demand for data exceeds supply in a very specific sense.
Start with demand. To make this digestible, let's confine our attention to US households in an urban or suburban area, which produces the majority of data traffic.
No simple generalization can characterize all users and uses. The typical household today uses data for a wide variety of purposes-email, video, passive browsing, music videos, streaming of movies, and e-commerce. Networks also interact with a wide variety of end devices-PCs, tablets, smartphones on local Wi-Fi, streaming to television, home video alarm systems, remote temperature control systems, and plenty more.
It is complicated, but two facts should be foremost in this discussion. First, a high fraction of traffic is video-anywhere from 60 to 80 percent, depending on the estimate. Second, demand peaks at night. Most users want to do more things after dinner, far more than any other time during the day.
Every network operator knows that demand for data will peak (predictably) between approximately 7 p.m. and 11 p.m. Yes, it is predictable. Every day of the week looks like every other, albeit with steady growth over time and with some occasional fluctuations for holidays and weather. The weekends don't look any different, by the way, except that the daytime has a bit more demand than during the week.
The bottom line: evenings require far greater capacity than other times of the day. If capacity is not adequate, it can manifest as a bottleneck at many different points in a network-in its backbone, in its interconnection points, or in its last mile nodes.
This is where engineering and economics can become tricky to explain (and to manage). Consider this metaphor (with apologies to network engineers): Metaphorically speaking, network congestion can resemble a bathtub backed up with water. The water might fail to drain because something is interfering with the mouth of the drain or there is a clog far down the pipes. So, too, congestion in a data network can arise from inadequate capacity close to the household or inadequate capacity somewhere in the infrastructure supporting delivery of data.
Numerous features inside a network can be responsible for congestion, and that shapes which set of households experience congestion most severely. Accordingly, numerous different investments can alleviate the congestion in specific places. A network could require a "splitting of nodes" or a "larger pipe" to support a content delivery network (CDN) or could require "more ports at the point of interconnection" between a particular backbone provider and the network.
As it turns out, despite that complexity, we live in an era in which bottlenecks arise most often in the last mile, which ISPs build and operate. That simplifies the economics: Once an ISP builds and optimizes a network to meet maximum local demand at peak hours, then that same capacity will be able to meet lower demand the rest of the day. Similarly, high capacity can also address lower levels of peak demand on any other day.
Think of the economics this way. An awesome network, with extraordinary capacity optimized to its users, will alleviate congestion at most households on virtually every day of the week, except the most extraordinary. Accordingly, as the network becomes less than awesome with less capacity, it will generate a number of (predictable) days of peak demand with severe congestion throughout the entire peak time period at more households. The logic carries through: the less awesome the network, the greater the number of households who experience those moments of severe congestion, and the greater the frequency.
That provides a way to translate many network engineering benchmarks-such as the percentage of packet loss. More packet loss correlates with more congestion, and that corresponds with a larger number of moments when some household experiences poor service.
Tradeoffs and Externalities
Not all market participants react to congestion in the same way. Let's first focus on the gazillion Web firms that supply the content. They watch this situation with a wary eye, and it's no wonder. Many third-party services, such as those streaming video, deliver a higher-quality experience to users whose network suffers less congestion.
Many content providers invest to alleviate congestion. Some invest in compression software and superior webpage design, which loads in ways that speeds up the user experience. Some buy CDN services to speed delivery of their data. Some of the largest content firms, such as YouTube, Google, Netflix, and Facebook, build their own CDN services to improve delivery.
Next, focus on ISPs. They react with various investment and pricing strategies. At one extreme, some ISPs have chosen to save money by investing conservatively, and they suffer the complaints of users. At the other extreme, some ISPs build a premium network, then charge premium prices for the best services.
There are two good reasons for that variety. First, ISPs differ in their rates of capital investment. Partly this is due to investment costs, which vary greatly with density, topography, and local government relations. Rates of investment tend to be inherited from long histories, sometimes as a product of decisions made many years ago, which accumulated over time. These commitments can change, but generally don't, because investors watch capital commitments and react strongly to any departure from history.
The second reason is more subtle. ISPs take different approaches to raising revenue per household, and this results in (effectively) different relationships with banks and stockholders, and, de facto, different budgets for investment. Where does the difference in revenue come from? For one, competitive conditions and market power differ across neighborhoods. In addition, ISPs use different pricing strategies, taking substantially different approaches to discounts, tiered pricing structures, data cap policies, bundled contract offerings, and nuisance fees.
The use of tiers tends to grab attention in public discussion. ISPs segment their users. Higher tiers bring more bandwidth to a household. All else equal, households with higher tiers experience less congestion at peak moments.
Investors like tiers because they don't obligate ISPs to offer unlimited service and, in the long run, raise revenue without additional costs. Users have a more mixed reaction. Light users like the lower prices of lower tiers, and appreciate saving money for doing little other than email and static browsing.
In contrast, heavy users perceive that they pay extra to receive the bandwidth that the ISP used to supply as a default.
ISPs cannot win for losing. The archetypical conservative ISP invests adequately to relieve congestion some of the time, but not all of the time. Its management then must face the occasional phone calls of its users, which they stymie with phone trees that make service calls last 45 minutes. Even if users like the low prices, they find the service and reliability quite irritating.
The archetypical aggressive ISP, in contrast, achieves a high-quality network, which relieves severe congestion much of the time. Yet, such firms (typically) find clever ways to pile on fees, and know how to stymie user complaints with a different type of phone tree that makes calls last 45 minutes. Even when users like the quality, the aggressive pricing practices tend to be quite irritating.
One last note: It is a complicated situation where ISPs interconnect with content providers. Multiple parties must invest, and the situations involve many supplier interests and strategic contingencies.
Some observers have alleged that the biggest ISPs have created congestion issues at interconnection points for purposes of gaining negotiating leverage. These are serious charges, and a certain amount of skepticism is warranted for any broad charge that lacks specifics.
Somebody ought to do a sober and detailed investigation to confront those theories with evidence. (I am just saying.)
What does basic economics tell us about congestion? Congestion is inevitable in a network with interlocking interests. When one part of the network has congestion, the rest of it catches a cold.
More to the point, growth in demand for data should continue to stress network capacity into the foreseeable future. Since not all ISPs will invest aggressively in the presence of congestion, some amount of congestion is inevitable. So, too, is a certain amount of irritation.
Copyright held by IEEE.
To view the printed essay, click here.
"... The 'Cloud' isn't magic, the 'Cloud' isn't fail-proof, the 'Cloud' requires hardware, software, networking, security, support and execution � just like anything else. ..."
"... Putting all of your eggs in one cloud, so to speak, no matter how much redundancy they say they have seems to be short-sighted in my opinion. ..."
"... you need to assume that all vendors will eventually have an issue like this that affects your overall uptime, brand and churn rate. ..."
"... Amazon's downtime is stratospherically high, and their prices are spectacularly inflated. Their ping times are terrible and they offer little that anyone else doesn't offer. Anyone holding them up as a good solution without an explanation has no idea what they're talking about. ..."
"... Nobody who has even a rudimentary best-practice hosting setup has been affected by the Amazon outage in any way other than a speed hit as their resources shift to a secondary center. ..."
"... Stop following the new-media goons around. They don't know what they're doing. There's a reason they're down twice a month and making excuses. ..."
"... Personally, I do not use a server for "mission critical" applications that I cannot physically kick. Failing that, a knowledgeable SysAdmin that I can kick. ..."
Disaster Recovery needs to be a primary objective when planning and implementing any IT project, outsourced or not. The 'Cloud' isn't magic, the 'Cloud' isn't fail-proof, the 'Cloud' requires hardware, software, networking, security, support and execution � just like anything else.
All the fancy marketing speak, recommendations and free trials, can't replace the need to do obsessive due diligence before trusting any provider no matter how big and awesome they may seem or what their marketing department promise.
- Why do Data Centers have UPS and Diesel Generators on-site? They know electricity can and does fail.
- Why do we buy servers will dual power supplies? We know they can and do fail.
- Why do we implement RAID? We know hard drives can and do fail.
Prepare for the worst, period.
Putting all of your eggs in one cloud, so to speak, no matter how much redundancy they say they have seems to be short-sighted in my opinion. If you are utilizing an MSP, HSP, CSP, IAAS, SAAS, PAAS, et all to attract/increase/fulfill a large percentage of your revenue or all of your revenue like many companies are doing nowadays then you need to assume that all vendors will eventually have an issue like this that affects your overall uptime, brand and churn rate. A blip here and there is tolerable.
Amazon's downtime is stratospherically high, and their prices are spectacularly inflated. Their ping times are terrible and they offer little that anyone else doesn't offer. Anyone holding them up as a good solution without an explanation has no idea what they're talking about.
The same hosting platform, as always, is preferred: dedicated boxes at geographically disparate and redundant locations, managed by different companies. That way when host 1 shits the bed, hosts 2 and 3 keep churning.
Nobody who has even a rudimentary best-practice hosting setup has been affected by the Amazon outage in any way other than a speed hit as their resources shift to a secondary center.
Stop following the new-media goons around. They don't know what they're doing. There's a reason they're down twice a month and making excuses.
Personally, I do not use a server for "mission critical" applications that I cannot physically kick. Failing that, a knowledgeable SysAdmin that I can kick.
MAT is an easy to use network enabled UNIX configuration and monitoring tool. It provides an integrated tool for many common system administration tasks, including Backups and Replication It includes a warning system for potential system problems, and graphing of many common system parameters. Click here for more.
Coming soon in version 0.24 will be an embedded interpreter with it you will be able to monitor any parameter you can write a script to capture. It also will create the ability to have OS specific configuration tools.
- Master System is a public-domain Unix systems configuration tool written in Perl. The system is architecture and operating system independent, but it can handle architecture and operating system dependent configuration. It is designed to control the configuration of large groups of systems that are configured in the same style, but are not necessarily identical. From a group at Rutgers University.
- Webmin is a free web-based admin interface for Unix systems. Via a web browser, you can configure DNS, Apache, Samba, filesystems, startup scripts, inetd, crontabs and more. Written in Perl5 and easily extendable. Supports several Linux versions and Solaris.
Conveniently Administering Remote Servers
What is edge computing and how it's changing the network Network World
Edge computing - Wikipedia
What's the difference between edge computing and cloud computing - Quora
Bandwidth alone won't solve application performance problems Network World
Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers : Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism : The Iron Law of Oligarchy : Libertarian Philosophy
War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda : SE quotes : Language Design and Programming Quotes : Random IT-related quotes : Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce : Bernard Shaw : Mark Twain Quotes
Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 : Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law
Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds : Larry Wall : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS : Programming Languages History : PL/1 : Simula 67 : C : History of GCC development : Scripting Languages : Perl history : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history
The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month : How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater�s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite
Most popular humor pages:
Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor
The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D
Copyright � 1996-2018 by Dr. Nikolai Bezroukov. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) in the author free time and without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.
FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.
This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...
The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.
Last modified: November 13, 2018