Why sites go down

Rhizomatica has recently moved into a phase of critical reflection about what we do. Specifically, we are looking to share not only our successes, but also the challenges we face, how the affect us and what we are trying to do about them. The idea and practice of local or alternative networks is not new, but is a space where we think there need to be some hard conversations about how to expand them and make them more resilient. This starts with being honest about our own shortcomings and the difficulties that exist when interfacing with the traditional or corporate infrastructure that we rely on to move some of our data or traffic around. So with that in mind, an explanation of why some sites die or go down for longer than a few days. Please note these are not in any particular order.


Reason 1. Political or Social problems

Rhizomatica has decided, for a host of reasons that are beyond the scope of this post, to work directly with organized communities and social organizations, rather than a more entrepreneurial or private investment approach. This means that while we are protected from some things such as vandalism (much more likely someone will steal something belonging to someone else than to themselves) it puts the project at the whim of local political issues. We have one site that has been down for 2 or 3 months due to a disagreement between two political factions within the town. This has meant that the administration computer was held “hostage” and no users could pay their monthly fee, buy credit, etc. and that the town stopped paying it’s Internet bill and therefore had no long-distance service. While the places we work are generally quite organized, these types of political disputes do happen and are pretty much impossible to do anything about from our end. Mainly we just have to wait for things to work themselves out locally.


Reason 2. Existing networks, “Interference” and Colonialism

We find that our service struggles when there is already a corporate telco network present or one comes along in a place where there is a community network. We’ve found there are a few explanations for this. The first is an issue around how people experience their phone and network and are conditioned around what traditional providers provide. The second is something more ephemeral relating to how people view development, modernity, progress and so on. This mentality is in turn shaped by the history of colonialism in Mexico. What this all means for our work is that we face, in some localities, expectations of what the community cell phone service should be like. So even if it’s vastly cheaper and more or less of the same quality, the fact that we don’t have direct interconnect and users have to dial using the country code or folks calling in hit a virtual exchange and then need to enter an extension to reach their destination, makes it so that some people prefer to use the more expensive, corporate network. Sometimes its just pure brand recognition. For some rural towns being able to show off that you can afford to use the corporate network holds prestige.

Another related issue we face is the existence of fixed wireless terminals; a fancy name for a cell phone that looks and operates as if it were a fixed line phone. Since coverage is scarce in many rural areas, people resort to buying these and attaching high gain directional antennas in order to pick up signals from really far away, behind mountains and so on. Strictly speaking this isn’t legal, but who can blame folks for doing what’s necessary in order to communicate? The problem for us comes in when the community installs a network and the fixed wireless terminal tries to negotiate between the community network right there and the weak signal coming from far, far away. The result is that the weak signal sometimes seems to disappear as far as the phone is concerned and therefore it can’t be used. This angers lots of people and has led to us having to shut down two networks permanently. If you understand Spanish, you can read more about it here>> http://wiki.rhizomatica.org/index.php/Site_selection


Reason 3. Infrastructure problems.

This is something that we’ve spoken about before and refers mainly to electricity and Internet. As a hybrid approach to networking, we rely on the existence of other systems to do what we do. This is certainly a vulnerability. It also makes what we offer much less expensive. To the point, we have had equipment damaged by electrical surges, or the whole system will go off when the power is out, which can sometimes last for days. The situation with the Internet is more or less the same, with the added issue that, at least in Oaxaca, the major routes to the IXP and the US seem to be constantly broken or not operating to any reasonable standard. This means it is really hard to connect long distance calls for the users. The solution to the electricity issues in most cases is having a backup battery and protecting the equipment from surges and discharges. But fixing the Internet is hard. Maybe we’ll end up trenching our own fiber to Mexico City :)


Reason 4. Damage from lightning

This plagues pretty much every rural network and relates partially to Reason 3, when the bolt strikes the power lines rather than the tower or installation directly. For direct strikes, we are working to find a cheap and reliable way to protect everything we put on the tower and everything that it connects to. It’ll add about 5 – 10% of the total installation cost, which is reasonable.


Some of the above can be fixed with a little more investment. Some are complex, deeper issues that we can’t fix on our own. And yet others are tied to the way we have and would like to continue doing things because they align with our values as an organization. We want to offer free/open, inexpensive communications options to people that allow them to both organize collectively and strengthen existing community institutions. We probably focused too much on getting costs down and not enough on making the networks robust. So now we can adjust and bring things into better balance.

If you haven’t seen it already, check out our friend Steve Song’s blog post about the potential of new approaches to cellular.

We are particularly excited about the fact that the new spectrum rules were published in April and we appear in the Federal Gazette equivalent as the reason for the new law.

Dealing with technical issues

We have 19 networks up and running around Oaxaca now and have decided to take a pause in regards to new installations in order to focus on tightening up our support infrastructure and processes to actually be able to attend to the issues that arise.

With all these networks running, we have zeroed in on some important points of failure.

First and foremost, the FLOSS GSM stack we use has some kinks that need to be worked out, which we are doing together with Holger Freyther from Sysmocom/Osmocom. These pertain mainly to paging problems, broken channels, and a big one that is perceived by user around how the audio is handled. Beyond that our hardware setup has been really stable in almost all cases, excepting one major ligthening strike that fried what we had in place to protect the BTS and also part of the BTS itself.

More generally, what we are seeing with regards to problems are more related to the reliability of the Internet we use and the electrical grid. From the Internet side, no matter with WISP we are working with, we are finding that there is congestion at the fiber that leaves Oaxaca, and consequently the VoIP aspect of what we do becomes impossible for some minutes or even hours every day. This issue, like the electrical grid issue, are related to major, national deficiencies. This is frustrating as it feels like there isn’t much we can do about it. We have begun a process at looking at other fiber options in Oaxaca with the government, but that is going slowly.

So much going on!

Some big pieces of news, lately.

The first one is that we just installed our 10th site. I am not sure if any of the Rhizomatica team thought we’d actually get to this point. When we started out community cellular we just an idea. There wasn’t really any afforsable hardware that worked well, the software had all kinds of bugs, and so on. We still have lots of challenges to overcome together with our partners and suppliers, but we received a huge piece of news at year’s end that seems to point to the continued existence and expansion of community cellular, at least here in Mexico.

Which brings us to the second big thing, one that has been the result of months, if not years of work by Rhizomatica, and lots of good timing. The regulator here in Mexico, IFETEL, just published it’s National Frequency Attribution Plan and:

“For the first time specific bands have been assigned for social use in the telecommunications sector. As part of this, various portions that are available within the segment known as the cellular band, between 824-849 and between 869-894 MHz, are now available for concessioning.

It is proposed that these portions of the spectrum are to be concessioned for the provision of rural connectivity, which could meet the immediate needs of basic telephone service in regions not served by existing licensees.”

This means that the entire country, minus Mexico City and the surrounding states, can now benefit from community networks like the one’s Rhizomatica helps to operate. This is huge and unprecedented, as far as we know.

And for all of you Spanish speakers, here is a video for your enjoyment.

GPRS is Go

GPRS testFirst data transmission tests for our GSM networks


Check out the video of our presentation at HOPE X in NYC here

Thanks to everyone for coming to our talk and all the kind comments and support we received

Movistar’s rural franchising model shows its true colors

Many months back we wrote about Movistar/Telefonica’s rural franchising scheme. Well it has become a reality, and it seems the local franchisee in Oaxaca is intent on following us around and attempting to install wherever we have already established our networks. While on the one hand this is a positive step – these communities have been practically begging to be connected for over a decade, on the other hand it represents the worst aspects of business-as-usual with regards to this and other telecom companies.

There are two aspects to what is going on in the Sierra that are most objectionable. The first is that the community must pay for all the upfront capital costs of setting up the network in order to have the “opportunity” to pay the exorbitant rates the incumbent providers charge per minute. The second aspect is that, at least with regards to the Movistar franchisee in Oaxaca, there is a very clear intention to descredit the work of Rhizomatica and our community partners by claiming we are operating pirate networks.

Here is what is going on: the Movistar franchisee has partnered with a local politician, and the two, it seems, have started some sort of shady business venture in which the politician puts up a small part of the money and convinces the villages to put up much more (around $40,000 USD). Then, perhaps, the franchisee and the politician split up the money, possibly greasing the palms of some of the local authorities. In order to convince villages to spend this amount of money, the local politician actively defames us and our model, which costs about 20% of the total cost of the Movistar option. All of this is nothing new. What is unfortunate is how the franchise model has been totally distorted and is being used to extract money from the local economy, rather than sharing a part of the money generated by the traffic, as originally proposed.

Oswaldo Martinez, the coordinator of Fundacion Santa Maria, Rhizomatica’s partner in Santa Maria Yaviche, Oaxaca, was recently interviewed by the state’s most important newspaper about this issue and stated, “it’s good that [the telecom companies] are coming, provided they respect us and vice versa, because we are competing legally. We want people to decide which of the two options to opt for based on a comparison of quality and costs … people who have money, go ahead and spend it, but we will continue with our project”.

According to Movistar’s own documents regarding the franchising model, the local operator keeps around 40% of the profit from the traffic, under the assumption that they (the local operator) have made a capital investment to put up the network. What is happening in the Sierra is that this capital or investment costs are being passed on to the communities themselves and none of the profit is being shared with them. Instead, it is all taken by the franchisee, who risked nothing. We have heard that America Movil/Telcel, the country’s largest, causi-monopoly operator is doing something similar in rural areas, requiring investment by the community of up to $90,000 USD (1.2 million pesos) in order to have service. Although a jaw-dropping figure, at least they don’t pretend to have any intention of sharing profits with the communities.

Speaking about the entrance of the large telecom companies under these new schemes, Oswaldo offered the following: “unfortunately, for our indigenous communities, the looting continues. A few are getting rich while the rest of us pay expensive service costs. In the case of phone calls, they are very expensive, but now we understand that there is a difference between what the service is really worth and the price we pay. A real and affordable rate is our right”.

The issue is not only economic. This situation is also telling of the way that these multinational corporations operate and their abhorance of competition and, god-forbid, conceiving of telecommunications as anything other than a money-making venture. Rhizomatica decided to work in the Sierra Juarez because we feel confident that the communities can resist the logic of capital and the way telecommunications and other services attempt to introduce this logic.

Oswaldo’s words are important in this respect. “What we seek is to restore autonomy to the people; that the State give us … concessions to operate our own communication services … The community telephone network supports our organization’s ability to generate alternatives … and share options that apply knowledge to [increase the quality of] daily life.”