Tuesday, November 27, 2018

Wormhole Musings

I have questions about how wormhole portals in science fiction stories work.

Recently I started reading another science fiction novel where wormholes allow instantaneous travel between distant points. In books that use this mechanism, the author typically explores how the ability to travel easily and quickly between the stars shapes the course of history.

But I always get hung up thinking about all the other ways in which a portal might possibly be used, for good or evil, in ways much less grand but potentially more disruptive than distant travel. Of course, since the use of wormholes in these books does not rely on our currently generally accepted science, these questions do not have well-defined answers. That's why I muse.

In this article, I ask some questions about how some of our currently accepted principles of physics apply (or don't apply) to wormholes, and ponder the ways in which one might use (or misuse) a wormhole based on the answer to those questions.

Caveat lector: If you want to keep reading wormhole stories without being distracted by questions like these, you might want to stop reading now. Because once you read these questions, you won't be able to unread them.

My Questions

How big and expensive is the equipment required to create and maintain a wormhole?

Mainly what I want to know for this question is whether the equipment is small and inexpensive enough that an individual can own one. If they are within the reach of many people, that makes it much more likely that there will be some people who will use it for unexpected purposes.

I once read a story in which someone had invented a personal flying belt that anyone could get for five dollars. With such easy personal mobility, border control suddenly became much more difficult, which of course led to some interesting problems. If anyone could buy and control a wormhole for five dollars, that would be a very different situation than if there were only a few wormholes controlled by a few rich and powerful entities.

How much energy is required to create and maintain a wormhole?

Although science fiction wormholes don't rely on any currently known physics, my feeling is that any scientifically plausible mechanism for a wormhole would require a prohibitive amount of energy to use. And I mean the word prohibitive literally: the amount of energy required would be so high, it would effectively prohibit the possibility of using a wormhole.

Since that doesn't make for good science fiction stories, we have to assume that the energy requirement is modest enough that we are able to produce and use wormholes. The question then becomes, how much energy is required? This question is related to the earlier question about cost, in that if a wormhole requires a relatively large amount of energy to operate, that could restrict its operation to a small number of controlling entities. Whereas if I can run it with a D-cell battery, there would be many more interesting things I could do with it.

It may not matter how much energy it requires to operate a wormhole, because, as discussed in some of my comments below, it seems likely that once you have a wormhole you could get as much free energy as you want.

What shape is a wormhole portal?

In most stories, wormholes portals are portrayed as circular areas that you step through, much like the entrance to a common tunnel. This is very convenient for imagining things like train lines that run through wormholes, and for thinking about the equipment that might be required to hold open a wormhole portal. That equipment is sometimes described as a torus with massive structures around it.

I think it is more likely that a wormhole would be spherical. You could enter it from any direction, and you would exit in a direction based on the direction you entered. This is a bit harder to visualize, which may be one reason it is not often described this way.

If a wormhole portal is a sphere, how does that impact the equipment required to maintain it? It would be tough to have equipment symmetrically on all sides and still have something that allows easy access. But maybe it doesn't all have to be completely symmetrical, so you can leave a few holes to let the trains get through the equipment so they can enter the portal.

Can I make a wormhole as large or as small as I want?

In most stories, wormholes are of a size that makes them convenient to step through, or drive a car or train through. Is this an essential feature of wormholes, or is it just that that happens to be the most convenient size? Could we make them any size if we wanted to? Perhaps big wormholes would be harder, but I would think smaller wormholes would actually be easier to make. And I can think of lots of interesting uses for small wormholes, depending on the answers to the other questions.

One example of a good use for a tiny wormhole would be to shine a laser through it and have a high capacity communication channel.

How do you control the location of the wormhole portals?

Some stories postulate that maintaining a wormhole portal requires physical equipment at both ends. In this case, the question of how to control the location of the portal is clear: you have to move the equipment to move the portal.

In other stories, the two ends of the wormhole are created at one location, after which one end can be moved to another location. In considering the geometry of wormholes, I would guess that it is possible to move one end of a wormhole through another wormhole, but perhaps only if the wormhole being transported is sufficiently smaller than the one it is being moved through.

If equipment is required at both ends of the wormhole, establishing a wormhole from A to B requires first traveling from A to B through normal space to deliver the necessary equipment, or possibly from C to B if the two ends of the wormhole don't need to be created in one place. This constrains the expansion of an interstellar civilization to the speed of light, which is annoyingly slow to some authors.

The more interesting case, as postulated in some stories, is that you can project the other end of the wormhole to a desired location without first having to get there some other way. This is, of course, a much-preferred mechanism if you want to quickly expand your network of gates, since who wants to wait many years while the slowship takes your gate to the next star? But what could we do if we could project the other end of our portal to anywhere we wanted in space?

If I can project tiny wormholes, I could do cut-less surgery. Mining would be much cheaper, as I could just project a wormhole down to where the ore is without having to tunnel or strip-mine down to it. I could make a great vacuum pump by putting one end out in space.

At a more banal level, I could eat as much as I want and not gain weight. I just need to project a tiny wormhole into my stomach and remove the food I just ate before my body digests it. I get all the pleasure of eating without suffering the problems of obesity.

I read one story in which a little wormhole was located on the bottom of a drinking glass, with the other end at the bottom of a vat of beer, wine, or whatever drink was selected. Each time the glass was set down, the wormhole would open to fill the glass, then close once the glass was full.

If I put on my black hat, the most obvious nefarious deed is, I project the other end of my wormhole into a bank vault and walk off with the cash. Or into a collection of classified documents and walk off with the secret plans. Or into my enemy's bedroom and kidnap him or kill him. I really only need to project a tiny wormhole, big enough for a bullet, to do a dastardly deed. Or so small it's only big enough for a packet of viruses that I inject into his bloodstream without him even knowing it.

If we can project one end of our wormhole to any desired location in space, perhaps we could project both ends. This would allow us to establish a wormhole between any two points anywhere in space, without having to have equipment at either end. This could actually be an interesting premise for a story, as it would allow for the case where there is a single wormhole-generating facility that creates all of the wormholes used throughout the civilization. That facility would presumably be controlled by some now-very-powerful entity, and would be both heavily secured and heavily attacked, so there are lots of opportunities for story lines.

The ability to create a wormhole between any two other points in space also opens up lots of additional opportunities for mischief. One could create a pretty effective weapon of mass destruction by creating a wormhole with one end in the middle of the sun and the other end where you want the destruction. Or put one end in the middle of a magma reservoir, or deep in the ocean, depending on the type of destruction desired. Or put one end in space to suck everything into the vacuum.

On the positive side, one could create a really nice package delivery system. Open a wormhole between the package source and destination, drop the package in for instant delivery, and close the wormhole.

Assuming we have the ability to create a wormhole portal anywhere in space, there is still the question of how we figure out where it gets created. Do we have to use trial and error to place the wormhole in just the right place? If we are trying to create a wormhole portal in a distant location, do we have to worry about the precision of our equipment, in the same way that launching a spaceship to land on Mars requires more precise equipment than launching one to land on the moon? Can we create the remote wormhole portal and then move it around at will, and if so, can we move it faster than the speed of light?

Is energy conserved when traversing a wormhole?

In most wormhole stories, one can step through a wormhole to get from one end to the other with no more effort than walking across the room. There is no explicit discussion of conservation of energy, and my assumption is that the authors don't worry about it because that detail doesn't advance the story. But I worry about it.

If I open a wormhole between Earth and its moon, there is a pretty big difference in the gravitational potential energy between those two points. When I want to put something in the wormhole portal on Earth and have it come out on the moon, do I need to supply the difference in energy between those two points? That would mean supplying a whole lot of energy to move in that direction. Conversely, if I step through the wormhole from the moon back to the Earth, what happens to all that gravitational potential energy?

If I can move from one end of a wormhole to the other end without having to supply that extra energy, then I can get free energy. Here's one way: go find a big dam with a hydro generating plant and install a wormhole with the entrance portal under the water at the bottom of the dam, just past the outflow of the generator, and with the exit portal just above the surface of the lake at the top of the dam. Since the entrance portal is underwater and the exit is above, water flows into the entrance portal and comes out at the exit portal. Thus the lake is ever refilled and our hydroelectric generators can keep running.

Maybe the wormhole technology works like a battery with regenerative braking on electric cars: it supplies the energy needed when traveling in one direction, and absorbs the excess energy when traveling in the other direction.

Is momentum conserved when traversing a wormhole?

If I am in New York City, the Earth's rotation is moving me at about 700 miles per hour relative to the center of the Earth. At the same time, Sydney is also moving at about 700 miles per hour, but in roughly the opposite direction, as it is almost on the opposite side of the Earth. If I open a wormhole between New York City and Sydney, and I step through, what happens to that 1400 miles per hour difference? Do I splat into the nearest wall at supersonic speed, or do I casually step through and continue walking to my destination?

If momentum is conserved, then I would be moving at a high speed relative to the exit point of the wormhole. If I put the appropriate mechanical devices next to the wormhole exit, I could send through a rock, catch it moving at 1400 miles per hour, and convert that kinetic energy to electricity. Then I could toss the rock back and do the same thing on the other side. Free energy.

The question of conservation of momentum is subtler than it first appears. If I want to conserve momentum, I come out of the wormhole in Sydney with that supersonic velocity relative to the city. But what does that mean for the angular momentum of the system? If I just moved that mass over to a new location and nothing else changed, then I have changed the angular momentum of the system. If the whole earth moves a tiny bit in the other direction, to keep the same center of mass, that could take care of that issue, but why should the whole Earth move when I use a wormhole? Would that happen if I were in an airplane? In a spaceship in low orbit? In a spaceship in high orbit? In a spaceship at the orbit of the moon, or beyond?

As with conservation of energy, perhaps the wormhole portals absorb or supply momentum as needed, transferring it to the surrounding masses. This could mean that wormhole portals would most effectively be placed on large masses such that they had a reservoir of momentum to transfer to or from. The larger the masses that were transferred through a wormhole, and the larger the relative velocity of the portals, the more momentum would have to be transferred, and the larger the attached mass would have to be.

How do physical forces propagate through a wormhole?

In every wormhole story I have read, light traverses a wormhole with no problems. I assume that means all forms of electromagnetic radiation traverse a wormhole equally easily. This presents another opportunity for a good energy source: put a wormhole portal in close orbit around the sun, then put the other wormhole portal on Earth. Stream that high-intensity light through and use it to drive solar cells for direct production of electricity, or as a heat source for standard steam turbines. If no equipment is required at the solar end of the wormhole, you're all set. If equipment is required, you might have to build some kind of refrigerator that brings that heat back to Earth and keeps the equipment cool.

How about gravity? How does that propagate through a wormhole? Most wormhole stories I have read describe travelers stepping through a wormhole and experiencing a discontinuity in the gravity field, meaning gravity is not propagating through the wormhole. This seems odd to me. Why would light propagate through a wormhole but not gravity?

The intensity of light from a point source drops off proportionally to the distance squared, which makes sense because the light is spreading out at that rate, and a fixed-size object intercepting the light will thus get less of it when it is further away. Because of this behavior, it makes sense to me that the amount of light that would come through a wormhole would be proportional to its size. If the wormhole is very small, only a small amount of light would come through.

Gravity also drops off proportionally to the distance squared, but not quite for the same reason. Given a particular mass, the gravitational force on that mass is independent of whether it is small and dense, or larger and less dense. The amount of area covered by the mass is not important, only its mass and its distance from another mass. If there is a tiny wormhole and I can measure a distance through that wormhole from my object to a large mass, wouldn't that mean the gravitational force is proportional to the square of that distance?

If gravity does propagate through a wormhole, perhaps I could make a null-gravity region by creating a pair of wormhole portals, then putting each one slightly above the surface of the Earth and upside down from each other. If you were to stand under one portal and look up, you would see the Earth above you. You have one Earth gravity below you and one above, so they cancel out and you have no gravity. A nice tourist attraction. Then again, the two Earths would also be exerting a gravitational pull on each other, so whatever is holding up each wormhole portal might be carrying the weight of the world.

On the other hand, given that General Relativity says that mass causes curvature of space, and thus gravity, and wormholes are usually described as some way of warping space, that seems to imply that being able to control wormholes means being able to control the curvature of space and thus being able to control gravity. So perhaps based on that we can choose how we want gravity to propagate through wormholes for our stories.

If you can turn wormholes on and off at will, you might be able to use this effect to get some free energy. You turn on a wormhole, have it pull up a weight, then turn it off, let the weight fall, and use that to generate energy.

What is the geometry of the wormhole connection?

A wormhole is usually described as a connection that goes through a higher dimension than the three dimensions in which we live. Those higher dimensions may present degrees of freedom that can lead to some curious and unpleasant results. Let me try to explain with a flatland analogy.

If I live in a two dimensional space, I can create a wormhole by folding that sheet of space until two points meet, then punching out a circle around those two points, and sewing those two circles together. This is topologically equivalent to attaching a hose that stretches up from a circle around one of those points and comes down at a circle around the other, with the assumption that the hose represents no distance (or a very short distance). A 2D creature could move from regular space onto the surface of that hose (assuming the hose diameter is much larger than the creature), then to regular space on the other end, then return to its original location via regular space, and all is well.

Now consider what happens if I take that same hose, but instead of going up from the first point and down at the second, I go up from the first point, then go around to the under side of the plane (which I can do without going through the plane if I have yet another dimension) and come up from the bottom side of the plane to meet the second point. Consider again what happens to that 2D creature who travels into the wormhole, out the other end, and returns to its starting point in normal 2D space. The result is that it comes back inverted. What was left is now right, and vice-versa.

I once read an old science fiction story in which there was a place deep within the Amazon where, if you navigated a certain course, it would reverse everything left to right. An enterprising businessman heard this and figured he could more efficiently make shoes by manufacturing only left shoes, then shipping half of them around this circuit, so he went exploring to find it. After going around the course, he looked at his sample left shoes, but they were all still left. Frustrated, he threw them all away, destroyed the worthless maps, and returned to civilization - only to discover that in fact the trick had worked, but he had not recognized it because he, too, had been reversed. But he could never find the place again.

Getting your body flipped left to right would probably be fatal. Almost all of our body chemistry is chiral, so you would not be able to extract any nutrition from most foods, and you would starve to death or die of malnutrition.

If there is an extra dimension in which a wormhole exists, why not two extra dimensions? If there are two or more extra dimensions, you now have the issue described above, and you will need to make sure you get the two ends of your wormhole attached with the right geometry, or things that move through the wormhole might not come out quite as expected.

Of course, a black-hat could surely come up with evil things that could be done with that kind of wormhole.

When considering wormhole geometry, another potential problem is the curvature of space in the wormhole. According to Einstein's Theory of General Relativity, curved space causes uneven acceleration. Too much curvature can lead to disastrous gravitational tidal effects that can tear things apart. Small wormholes would be most likely to have this problem. Larger wormholes, like South Pass through the Rockies, would allow that curvature to be spread out enough to be hardly noticeable.

In what reference frame is traversal of the wormhole instantaneous?

This is the issue which to me is the killer.

Einstein's Theory of Special Relativity is quite well supported by experimental evidence. According to that theory, there is no such thing as universal simultaneity, so we have to ask what instantaneous travel means.

You may have heard that, according to Special Relativity, if observer A with clock A in spaceship A is moving near the speed of light relative to observer B, clock A will run more slowly than observer B's clock B, according to observer B, due to time dilation. But at the same time, according to observer A, observer B with clock B is moving near the speed of light relative to A, so observer A sees clock B as moving more slowly. This effect is the core of the twin paradox, where one twin gets on a spaceship from Earth, flies away at near light speed, and returns, while the other stays on Earth.

The twin paradox is resolved by noting that there is an asymmetry between the twins: one stays at rest on Earth, whereas the other accelerates three times during the trip (takeoff, turnaround, and landing). This difference is the key to understanding the paradox and determining that the twin on the spaceship ages more slowly than the one left on earth.

In 1971 a couple of scientists ran an experiment where they took some atomic clocks with them on commercial flights around the world and confirmed that they really did slow down as compared to the stationary atomic clocks left behind, just as predicted by Special Relativity (and by General Relativity, which predicted time dilation due to gravitational differences).

For instantaneous travel between wormholes, it seems like we can set up a symmetric situation so that we can't resolve our paradox the same way as for the twin paradox. Consider the situation where we have a wormhole between two spaceships (or planets, if you prefer) A and B that are moving at near the speed of light relative to each other. As noted above, the observer in each location observes the clock moving more slowly at the other location. If person C with clock C steps from spaceship A to B through the wormhole, spends a bit of time on spaceship B, then comes back to spaceship A, observer A will calculate that clock C will be behind clock A, having moved more slowly than clock A while it was on spaceship B. If person D with clock D steps from spaceship B to A through the wormhole, spends a bit of time on spaceship A, then goes back to spaceship B, observer A will calculate that clock D will be ahead of clock B, having moved more quickly than clock B while it was on spaceship A. But in this symmetric situation, observer B will calculate that clock C will be ahead of clock A, and clock D will be behind clock B, the opposite of what observer A calculates. So which is it?

The problem here is that statement that travel between wormholes is instantaneous. According to Special Relativity, two events that occur at the same time but different locations in one reference frame will occur at different times in a reference frame that is moving with respect to the first. For our example, this means that if observer A sees person C moving instantaneously through the wormhole from A to B, observer B does not see person C moving instantaneously through the wormhole except for when A and B are right next to each other. And since A and B are moving with respect to each other, they will not be right next to each other for at least one leg of the wormhole round trip. When A and B are not right next to each other, what appears as simultaneous in one reference frame is not simultaneous in the other reference frame.

The only way I know of that is consistent with Special Relativity that would allow wormhole travel to be instantaneous according to both ends of the wormhole would be to constrain wormholes to be stationary relative to each other. But this would be a pretty strong constraint for stories, since essentially everything in the universe is moving relative to each other, and even the rotation of a planet is enough velocity variation to cause measurable time issues across the kind of distances wormholes sometimes connect.

But wait, it gets crazier. By the laws of Special Relativity, if you have any mechanism that lets you move between two points faster than the speed of light, in any arbitrary frame of reference, you can use that mechanism to travel backwards in time. The Tachyonic antitelephone is an example of how being able to send a message faster than light allows sending a message backwards in time, and this same principle applies to sending an object rather than a message.

One way to explain this is based on the assertion of Special Relativity that two events that are not at the same location in space that occur simultaneously in a frame of reference A will not be simultaneous in a frame of reference B that is moving with respect to A. In frame B, one of those two events will happen before the other. Let's assume that we have a wormhole with a pair of distant portals that are stationary in frame A, and another wormhole with portals stationary in frame B, moving with respect to frame A in the direction from one of the A portals to the other. We arrange the portals such that wormhole portal B2 is immediately adjacent to wormhole portal A2 at the starting time of our experiment according to observer A located at A1, and we arrange that B1 and B2 are adjacent to A1 and A2, respectively, at the same time in frame B. At the starting time in A, we step from portal A1 to A2. Since we arranged for B2 to be adjacent to A2 at this time, we can immediately move over to B2 and step through to B1, which we assume is instantaneous in frame B. Because we have arranged that B1 is adjacent to A1 at the same moment as B2 is adjacent to A2 in frame B, when we exit B1 we can then hop back over to A1 and complete our circuit in space. Since our trip through the wormhole B is instantaneous in frame B, it will not be instantaneous in frame A. For the traveler, all four legs of the trip are nearly instantaneous, but for an observer who remains in A only three legs are, with the leg through wormhole B not being instantaneous. Depending on which direction travelers takes around this loop, they will return to A1 either well after or well before the time they left.

The amount of time is proportional to the distance traveled through the wormholes and is related to the velocity of one frame with respect to the other. If frame B is traveling near the speed of light relative to A, the amount of time will be close to the light-distance between the two ends of the portal, so even if you are "just" traveling to Proxima Centauri B near Alpha Centauri, the closest extrasolar star group to Earth at four light years away, you could travel up to four years into the future or the past. The effect is less pronounced, but still present, at lower speeds.

Note that Special Relativity itself doesn't preclude faster-than-light messages or travel, it just says that being able to do so allows sending a message or traveling backwards in time, as demonstrated above. Our current theories do not say this is not possible, but most people believe in causality and thus find time travel problematic.

If you want to get a better intuitive feel for some of the weird things that happen when you start moving at near the speed of light, check out the free video game A Slower Speed of Light from MIT.

Potential Answers

Given that typical science fiction wormholes are based on new science beyond our current theories, we have a lot of leeway in deciding how that science works so as to create the conditions that best advance our story. We could say that managing wormholes requires an amount of money and energy that are only available to large organizations, or we could say that, once the science is known, wormholes are easy and cheap and anybody can make them, and see what kind of havoc is wreaked. We could say that small wormholes are easy to make, or that larger wormholes are easier. We could choose the geometry of the wormhole and portals to be troublesome or trivial. We could say that wormhole portals require equipment to maintain, or that we can cast them anywhere with ease.

All of the above choices are pretty easy in the sense that they are about the fictional new wormhole science and don't conflict with our existing science. Things get a little harder when we try to decide how conservation of energy and momentum work with wormholes, but even there we should be able to postulate something that allows us to remain consistent with known science, such as the wormhole absorbing or supplying the difference, or perhaps even requiring an exchange of equal mass from either end of the wormhole.

Propagation of gravity through a wormhole seems to me a little more difficult to deal with. As mentioned above, you might be able to claim that wormhole technology allows controlling the curvature of space. But another view of mass and space is that mass is the curvature of space, in which case making space curve is equivalent to creating mass, and at that point we get into all the questions of conservation of mass and energy and where it comes from when curving space for a wormhole.

The one that I really can't figure out how to make consistent is, as mentioned above, the question of time. The main reason wormholes are typically introduced is to allow faster-than-light travel, which, as described above, is what leads directly to the potential of time travel, according to Special Relativity. For all of the other questions, it seems like it may be possible to define some new science that answers those questions in a way that does not require us to discard any of our current well-established scientific theories, but for faster-than-light travel, I don't see any way to do this.

I can't even just assume that Special Relativity doesn't apply in that universe. There is a deep connection between having the same laws of physics everywhere, electromagnetism, and having a maximum velocity for any matter or information. Special Relativity builds on the work of Newton and Maxwell. and discarding it would require some other significant changes to the way the universe works.

A science fiction author might choose to focus on how wormholes allow time travel, as Robert L. Forward does in some of his stories. For the other stories, the ones that don't mention time travel, I just have to suspend my understanding of Special Relativity and enjoy the story as told.

Friday, April 13, 2018

Golang Web Server Auth

An example of authentication and authorization in a simple web server written in go.

Contents

Background

As described in my previous blog post, I recently rewrote my image viewer desktop app as a web app, for which I wrote the web server in go.

Since I was adding a new potential attack vector, I wanted to add security; but since this is only available on my internal network, and it's not critically valuable data, I did not need enterprise-grade security. In this post I describe how I implemented a relatively simple authentication and authorization mechanism, in particular highlighting the features of go I used that made that easy to do. For a simple app such as this one, the third of the three As of security, auditing, can be done with simple logging if desired.

The code I present here is taken from the github repo for my mimsrv project, with links to specific commits and versions of various files. You can visit that project if you'd like to see more of the code than I present in this post.

Before Auth

Go has good support for writing simple web servers. The net.http package allows setting up a web server that routes requests based on path to specific functions. In the first commit for mimsrv, before there was any code for authentication or authorization, the http processing code looked like this:

In mimsrv.go:
func main() { ... mux := http.NewServeMux() ... mux.Handle("/api/", api.NewHandler(...)) ... log.Fatal(http.ListenAndServe(":8080", mux)) }
In api/api.go:
func NewHandler(c *Config) http.Handler { h := handler{config: c} mux := http.NewServeMux() mux.HandleFunc(h.apiPrefix("list"), h.list) mux.HandleFunc(h.apiPrefix("image"), h.image) mux.HandleFunc(h.apiPrefix("text"), h.text) return mux } func (h *handler) list(w http.ResponseWriter, r *http.Request) { ... }
The above two functions set up the routing and start the web server. The code in mimsrv.go creates a top-level router (mux) that routes any request with a path starting with "/api/" to the api handler that is created by the NewHandler function in api.go. The top-level router also defines routes for other top-level paths, such as "/ui/" for delivering the UI files.

The api code in turn sets up the second-level routing for all of the paths within /api (the h.apiPrefix function adds "/api/" to its argument). So when I make a request with the path /api/list, the main mux passes the request to the api mux, which then calls the h.list function.

Adding Authentication

To implement authentication in mimsrv, I added a new "auth" package with three files, and modified mimsrv.go to use that new auth package. The most interesting part of this change is that it implements the enforcement of the constraint that all requests to any path starting with "/api/" must be authenticated, yet I did not have to make any changes to any of the api code that services those requests.

When I originally wrote my request routing code, it could have been simpler if I had defined everything in one mux. I didn't do that because I think the approach I took provides better modularity, but in addition, that structure made it easy for me to require authentication for all of the api calls.

The authentication code itself is not trivial, but wiring that code into the request routing to enforce authentication for whole chunks of the request path space was. I wrote a wrapper function and inserted it in the middle of the request-handling flow for requests where I wanted to require authentication.

To wire in the authentication requirement for all requests starting with "/api/", I changed mimsrv.go to replace this line:
mux.Handle("/api/", api.NewHandler(...))
with these lines:
apiHandler := api.NewHandler(...)) mux.Handle("/api/", authHandler.RequireAuth(apiHandler))
Here is the RequireAuth method from the newly added auth.go:
func (h *Handler) RequireAuth(httpHandler http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request){ token := cookieValue(r, tokenCookieName) idstr := clientIdString(r) if isValidToken(token, idstr) { httpHandler.ServeHTTP(w, r) } else { // No token, or token is not valid http.Error(w, "Invalid token", http.StatusUnauthorized) } }) }
The RequireAuth function looks at a cookie to see if the user is currently logged in (which means the user has been authenticated). If so, RequireAuth calls the handler it was passed, which in this case is the one created by api.NewHandler. If not, then RequireAuth calls http.Error, which prevents the request from being fulfilled and instead returns an authorization error to the web caller. When the mimsrv client gets this error it displays a login dialog.

The other code I added handles things like login, logout, and cookie renewal and expiration, but all of that code other than RequireAuth is specific to my implementation of authentication. You could instead, for example, use OAuth to authenticate, in which case you would have a completely different mechanism for authenticating a user, but you could still use a function similar to RequireAuth and wire it in the same way.

Adding Authorization

Wrapping selected request paths as described above makes it so that authentication provides authorization for those requests. This coarse-grained authorization is a good start, but for mimsrv I wanted to be able to use fine-grained authorization as well. As this is a simple program with a very small number of users, I don't need anything sophisticated such as role-based authorization. I chose to implement a model in which I only define permissions for global actions, then assign those permissions directly to users.

For this simple permissions model, I needed to be able to define permissions, assign them to users, and check them at run-time before performing an action that requires authorization. My permissions are simple strings, stored in a column in the CSV file that defines my users. To give a permission to a user, I manually edit that CSV file, and to check for authorization before taking an action, the code looks for that permission string in the set of permissions for the current user.

The one piece that is not obvious is how to pass the user's permissions to the code that needs to check them. The reason this is not obvious is because the http routing package defines the function signature for the functions that process an http request, and that function signature includes only the request and a writer for the response. You can't simply add another argument in which you pass your user information, so you have to dig a little deeper to figure out how to pass along that information.

The solution relies on the fact that there is a Context attached to the Request that is passed to the handler function. By adding the user info to the Context, you can then extract that information further along in the processing when you need to check the permission.

The RequireAuth function validates that the user making the request is authenticated, so it already has information about who the user is, and this is the point at which we want to add the user info to the Context. We do this in our RequireAuth function by replacing this line:
httpHandler.ServeHTTP(w, r)
with these lines:
user := userFromToken(token) mimRequest := requestWithContextUser(r, user) httpHandler.ServeHTTP(w, mimRequest) func requestWithContextUser(r *http.Request, user *users.User) *http.Request { mimContext := context.WithValue(r.Context(), ctxUserKey, user) return r.WithContext(mimContext) }
When the code needs to know whether the current user is authorized for an action, it can call the new CurrentUser function, which retrieves the user info from the Context attached to the Request, from which the code can query the user's permissions:
func CurrentUser(r *http.Request) *users.User { v := r.Context().Value(ctxUserKey) if v == nil { return nil } return v.(*users.User) }

Summary

While implementing authentication and authorization in a web server takes more than just a few lines of code, at least the part about how it gets tied in to the http processing in go is only a few lines. Although that part is only a few lines of code, it took me a while to dig around and find exactly how to do that. I hope that this article can save some other people a bit of time when doing their own research on how to add auth to a go web server.

Tuesday, March 13, 2018

Golang server, Polymer Typescript client

Finally, a web development environment I enjoy using.

Contents

TL;DR

I have found Go to be a nice tool for developing a small web server, and Polymer + Typescript to be a nice combination for developing a web UI. The Go server acts as both the API server and the static content server delivering the UI pages. If you think you might want to try this approach, you can look at my mimsrv program on github as an example. If it looks too complicated, browse in the git history back to some of the earliest commits, such as the first ui commit and the first api commit, to see how things looked at a simpler time.

Background

I have been developing web pages and apps for a long time, since the earliest days of HTML when there were no tools more sophisticated than a text editor, and server-side scripts were the only form of executable web code. In 1994 I wrote htimp, an experiment in how to attach a web browser to an interactive program with a lifetime longer than a single message.

Over the years I tried many technologies, including JavaServer Pages, JavaServer Faces, PHP, jQuery, and others I have forgotten. Some were better than others (more accurately, some were bad and some were excruciating), but I never felt any of them provided a reasonable mental model for how to put together an application.

I was away from the web UI scene for a while, and when I got back to doing some web development a couple of years ago, things seemed to have improved quite a bit. In the last year, I have been introduced to a few technologies that, in combination, provide me with a development environment with a working mental model of how to put together a program, and a set of tools that makes it easy to do that at a good clip.

The three technologies that together have brought pleasure back to my web programming are:
  1. The Go language and development environment
  2. The Typescript language
  3. Polymer-2 (and Web Components) with decorators
Below I describe the project on which I tried out these technologies, followed by a discussion of what I liked about them.

Mimsrv

Mimsrv is a web server and UI to view a collection of photos. It is a replacement for mimprint, which is a desktop app that I originally wrote starting in 2001 in Java, and converted to Scala starting in 2008.

A couple of years ago I started looking into rewriting mimprint once again, this time as a web app. As a web app, I would no longer have to worry about distributing a desktop application to the various machines I have on which I wanted to view my photos. I also thought I should be able to leverage the web browser's media capabilities so that I would not have to develop or support that whole chunk of code.

The tools I tried were never nice enough to pull me in and get me going on that replacement, and I had moved my rewrite-mimprint project way down on my TODO-list.

At Google last year I worked on the open-source Datalab project. When I started on it, we were using jQuery and Javascript. I liked it when we converted to Polymer-2 and Typescript, and I liked it more when we switched to using Polymer decorators.

I started learning Go in order to review code from my teammates. It took a little getting used to, but the more I learned, the more it made sense to me. I felt it was much easier to understand the existing Go codebase than similar codebases I had looked at in other languages. It grew on me, and after I started adding my own Go code to the project, I was surprised at how much I liked using it, and I felt that I was making pretty good coding progress.

I thought the combination of Go for the server, and Polymer and Typescript with decorators for the client, worked quite well, and I decided to try it for my personal project. So far that combination has worked well for me, and I have been quite happy with it.

What I Like

Offline Development

One of my requirements is that I be able to develop when I am offline. I insist on this because one of the situations in which I have the most amount of time available for programming on my personal projects is when I am traveling and often don't have network access.

In a previous attempt at putting together a collection of technologies for developing web apps, some of the pieces used maven, and I was unable to figure out how to convince it not to go out looking for new versions of the snapshots I needed every time it compiled.

After using Go on a project at work and being pleasantly surprised at how much I enjoyed using it, I decided to see it if would work for my personal projects. When I downloaded and installed it, I was delighted to discover that, not only did the installation provide everything I needed to compile and run my programs, but it also included all of the documentation and the Go Tour, so those would all be available to me offline!

Similarly, the Typescript and Polymer tools allow just building the code, without attempting to do any dependency resolution, so can easily be used offline.

Simple Mental Model

There are a couple of changes to the web app landscape that have made for a much simpler mental model than in the old days. The main one is the Single Page Application (SPA). With the old approach of having to move to a new page every time the user took an action, saving state across those page changes required mental and technical gyrations. With a SPA, you make AJAX calls to the server using XMLHttpRequest, and just keep your state in variables as in any other program.

The SPA model also allows for a clean separation of responsibility between the server and the client. With Polymer, all of the UI manipulation is handled in the client, so the server doesn't need to deal with any kind of templating of client-side functionality. This means the server can focus on the API and on just delivering the UI code to the client, and the client can focus on managing the UI and making API calls.

The other big change on the client side is the progress that has been made on the asynchronous programming model. At first we had to pass around success and failure callbacks, which requires splitting code up in unwieldy ways around every asynchronous call. The introduction of Promises provided a nice way to avoid the "callback hell" of deeply nested callbacks, but still requires chopping your code up around every asynchronous call. Lastly, the introduction of the async and await keywords made asynchronous programming almost as straightforward as synchronous programming. I'm particularly impressed that you can do things like have an if-statement with synchronous code on one side and asynchronous code on the other side, or a loop with an asynchronous call in it. This is so much simpler to reason about than if you had to figure out how to do that with callbacks or even Promises.

Simple Dependency Management

The few times I had to deal with Maven were unpleasant. I found it hard to control, hard to configure, and hard to understand what it was doing. Perhaps it's just that, with the march of time, people have figured out how to make dependency management better, but I found the dependency management in both Go and Polymer to be pleasant to use.

In Go, when you need a package, you just say go get package, and it downloads that package and all its dependencies. Assuming you follow the Go conventions when naming and locating your package, when someone then wants to download your package, they do the same thing, and Go will also download all of your dependencies to their system.

Polymer-2 uses bower for its package management, and it is almost as easy to use. The bower.json file lists the packages needed, and running bower install installs those packages and their dependencies. When you add a new dependency to one of your Polymer components, you just run bower install --save new-package to download that new package, and you're done. Not quite as effortless as go, but much better than my experience with maven.

For both Go and bower, they don't attempt to download anything except when you explicitly tell them to with go get or bower install, which is good for offline development.

Simple Compilation

For pretty much my whole programming career, I have been accustomed to using some kind of build tool that requires a configuration file: make, ant, maven, sbt, grunt, bazel, gradle, and others.

Go is different: it is so opinionated about where you have to put your packages and how you have to name stuff, that it has all the dependency information it needs by looking at the source files. You just tell it to build your program with the command go build, and it does it. No build config file required.

The Typescript compiler and Polymer build commands do require config files, but they were pretty simple to set up and understand, and seldom need to be modified. Running tsc compiles all the Typescript files to Javascript, and running polymer build packages all the Polymer Javascript and HTML files into a directory where they are served by the Go server.

Type Safety

I like the compiler to catch as many errors in my code as possible. Using compile-time types allows the compiler to spot more errors. This is why I greatly prefer Typescript over Javascript.

Go is also a compiled and typed language, so it catches a lot of problems before execution time.

Separation of Concerns

While I don't think having to use multiple languages is a benefit, the ability to select the best tools for different parts of the problem is. Go works very well as a web server for API calls and static content. Most people using Polymer embed Javascript code in their HTML file, but I prefer using Typescript and am happy putting that in a separate file from the HTML, where my editor understands it better.

Go http support

Go has a nice http package that makes it easy to define web routing and implement handler functions.

Because Go supports functions as first-class values, it's easy to define a function that can take a function as an argument and return another function. In my case, I used that approach to create a function that I could use to specify that certain parts of my API required authentication.

I wrote my http handlers to do only the marshaling and unmarshaling of data and then call the underlying routine that implements the requested functionality. This made it easy to write unit tests of the underlying function. But Go also provides a nice testing package for http handlers that makes it relatively easy to test the http handler as well.

Room for Improvement

I'm pretty happy with this collection of technologies, but there are a couple of things I would like to see improved.

Polymer/Typescript type mismatch

Polymer decorators are a nice improvement over the previous approach, as there is now much less boilerplate and repeated code. But I still have to specify a type in each Polymer.decorators.property line, and that type is not quite the same as the Typescript type (for example, string vs String, any vs Object).

I suppose this is not that surprising, given that Typescript is not officially supported by Polymer. I guess that's really what I would like to see happen.

Debugging Typescript

Writing Typescript rather than Javascript is nice, but when it gets loaded into the browser it's Javascript, so debugging in the browser uses the transpiled Javascript. The Javascript is usually close enough to the source Typescript that it's manageable, but it would be nice to be able to debug with the Typescript source code.

Maybe this situation will get better when WebAssembly gets implemented.

Thursday, June 22, 2017

FiOS - A Cautionary Tale

I delayed signing up for Frontier FiOS because I was concerned they might screw things up. I should have been more concerned.

This is a long post. Consider it entertainment. Or just skip to the Answers.

Contents

The Need for Speed

I have had internet connectivity for decades, starting back with modems so slow, I knew people who had to pause in their typing to let the modem catch up. I appreciated every doubling of speed as each generation of modem arrived. I was surprised when modem speeds reached 4800 and then 9600 baud - how could you get more bits per second than the 3K bandwidth of a phone line? - and I was astounded with the jump to 56K modems.

When DSL came out, I waited impatiently for it to be available in my neighborhood, and signed up as soon as I could. After years of using a 56K modem, my 740Kbps DSL line was satisfyingly fast.

I lived with 740Kbps for six years, until one day my DSL modem broke. While researching new modems, I learned that I could have my service switched from Frame Relay to ATM and bump up my speed to 3Mbps. Normally this would mean my service would be out for 10 days while they did that, but since it was already out, it seemed like a good time to make the switch. The speed bump from 740Kbps to 3Mbps was a mere 4x, far less than the 13x increase from the 56K modem to 740Kbps DSL, but still, 3Mbps was satisfyingly fast.

Verizon actually offered FiOS in my neighborhood fairly early, but I was pretty happy with my DSL service, and I wasn't doing anything that I thought needed more than 3Mbps bandwidth. I remained happy with my 3Mbps for over a decade. But technology marches on; I bought some HDTVs, started watching more YouTube, and started working from home more often using bandwidth-hungry remote desktop applications. My 3Mbps connection was not sufficient to stream HDTV movies and YouTube clips, and my remote desktop experience was annoyingly slow. I was finally feeling the bandwidth squeeze.

Still, I delayed upgrading to FiOS. I had heard that I would have to give up my copper-wire land lines, which I was not keen to do. Some years ago our power was out for over a week; batteries everywhere ran down - even the local cell towers ran out of juice after a few days, so there was no cell service in our neighborhood - but, with our copper wires, we had phone service the whole time. I liked that.

In addition, by this time Verizon had sold to Frontier, and based on my experience and anecdotes I read, I was concerned that Frontier would mess something up when dealing with my service change request, particularly since my situation was rather unusual.

My Unusual Situation

In my case, there were a number of things about my situation that gave me pause when thinking about asking for any kind of change.
  1. I have a land line. This has become increasingly rare, and it seems Frontier is deprioritizing phone service so they can focus on providing internet and television service. It seems they want to provide packages that include everything, or at least include both internet and television.
  2. Actually, I have two land lines. I'm not sure I know anybody else who has two land lines at home any more. I used to have three, but finally got rid of the third line after disposing of my last FAX machine years ago. Although I have two lines, they have not both shown up on my monthly bill for many years now. Oh, I am paying for two lines, it's just that the second line is not itemized anywhere. If you didn't know a-priori that I had two phone lines, you would hardly be able to tell that by looking at my phone bill. Based on conversations I have had with support and billing people at Frontier, it's not obvious to them either, although after I point it out, and with enough digging, some of them could figure it out.
  3. I had a DSL line. As I mentioned, I delayed for quite some time in switching from DSL to FiOS. The longer I delayed, the fewer people had DSL lines, and the less Frontier cared about them. For this particular problem, I suppose my delaying upgrading perhaps made things worse.
  4. My DSL service provider was not Frontier. This caused a fair amount of frustration any time I had a service issue with my DSL line.
My DSL service was perhaps the most unusual part of my situation. Back in 1999, when I originally ordered DSL, GTE (yes, it was that long ago!) had partnerships with Internet Service Providers who provided the actual internet service. These are known as CLECs (Competitive Local Exchange Carriers). So GTE provided the line, and my selected ISP provided the internet service. Originally I paid GTE directly for the line and I paid the ISP for the internet service. But when I switched from Frame Relay to ATM service, my billing also changed so that I paid everything to the CLEC, and they paid Verizon for the line.

Back then many of the carrier ISPs had annoying policies such as blocking some ports, so it was nice to be a customer of a smaller ISP that was more interested in making its customers happy. The downside was that, whenever there was a service problem, I had to deal with two companies, and they each tended to say it was the other company's problem.

By the time I was considering switching from DSL to FiOS this year, it had become perhaps comically bad: when I talked to support and billing people at Frontier, they were completely unaware that I had DSL service on my Frontier phone line, and even with a lot of digging, nobody I talked to at Frontier this year was ever able to find even a trace of information about my DSL line.

On the other side, my ISP had been acquired multiple times over the years, each time by a larger and more remote company, until by this year they were no longer in the DSL business and no longer in the residential ISP business. Somehow through all this, my residential DSL line kept working, but I did start to feel I was skating on ever-thinning ice.

It was time to take the plunge and upgrade.

Questions

Before ordering FiOS service, I wanted to get the answers to four questions:
  1. What service options do I have?
  2. What equipment will be installed and where?
  3. What is the installation process?
  4. How much will it cost?
How does one answer questions like these in today's world? Hit up the internet, of course.

Research

Frontier's Web Site

I started by browsing Frontier's web site looking for information about their service offerings.

NOTE: Frontier's offerings are regional, so you may see different web pages than what I describe.

Their FiOS page shows four levels of service: 50Mbps, 75Mbps, 100Mbps or 150Mbps. Since I wanted both internet and phone service, I headed over to the bundles page to see what I could get. I don't want their television service, so I unchecked the "Video" box. This shows three bundles: two that include 30Mbps internet service (30? that's not one of the speeds listed on their FiOS page!) and one that includes 50Mbps. Do they offer bundles that include internet service faster than 50Mbps? Their web site doesn't say.

Their Phone page shows me information about copper-line phone service (lower in the page it says "Our reliable copper power stays on even when the power goes out or in an emergency"), where they list two plans that differ by $3. Confusingly, this phone service - you know, POTS using analog signals on copper wires - is called "Digital Essentials." Are there any other optional add-ons? There are a fair number of features bundled with the basic phone service, and a lot more bundled with that extra $3, but is that it? Like, I currently have an unlisted phone number, what's the charge for that? Sorry, that kind of stuff is not on their web site. Ah, here on the the Digital Phone Unlimited page, it says "Optional international calling packages are available for great savings", so apparently there are other options available - but it's not a link, so I have no idea what kind of packages they might offer.

How about a second phone line, how much does that cost? Sorry, that's not on the web page. VoIP? Oh, maybe you mean FiOS Digital Voice. Beats me what the scoop is on that. If you go to Frontier's FiOS Bundles page, where it says the phone service in their bundles is Digital Phone Unlimited, and you click on the Learn More button for the phone service, it takes you to that phone page I mentioned above - you know, the one that says "Our reliable copper power stays on even when the power goes out or in an emergency." So, if I get a bundle that includes FiOS internet, does that bundle include Digital Phone Unlimited running on copper wires?

The details of the above web pages are what they look like now, in June 2017. I believe they have changed since I did my initial research a few months ago, but the gist is the same: I was unable to figure out what options were available to me by reading their web site.

Besides looking at Frontier's web pages, I did a lot of Googling and browsing of other web sites. I learned a lot in general about equipment, but it was hard to know how much of it would apply to my situation. Although I did not record the time I spent browsing Frontier's and others' web sites, I estimate it was probably about five hours.

It was time to move on to online chat to get more answers.

Online Chat

I had six online chats with Frontier, totaling about 4 1/2 hours. Between each chat I did more online research, looking for details about the equipment and the installation experience both inside and outside the house. It was difficult to get a good handle on these details, particularly since I sometimes got conflicting answers from the Frontier people I chatted with.

For example, one of my questions was whether I could keep my copper phone lines, or whether I would be required to switch my phone service to fiber. One of the people I chatted with said this:
You would have to switch to a digital phone service ! Voip. Which basically means Voice over Internet
Another one said I could keep my phone service on copper and get their "Simply FiOS" service, which is fiber with only internet service.

Ordering

Once I reached the point where I felt I had as good answers as I could get - which admittedly were not always very good - it was time to place my order.

On April 9 I called Frontier to place my order. I would say the fact that it took me well over an hour to place my order was the first hint of trouble, but in truth there were plenty of hints during the many chats I had, where I was not getting consistent answers.

Part of the reason the phone call took so long was due to my unusual situation. The DSL was not much of an issue during ordering, since it was completely invisible to them and they couldn't do anything about it. The real trouble was that second phone line. Figuring out how to deal with that took probably 45 minutes.

When I asked if I was required to switch my phone service from copper to fiber, the service rep first said no, but then went and asked someone else, came back and said yes, I would have to switch. I would have preferred to keep my phones on copper (and especially I would have preferred it given how much trouble I have had with the switch), but I was not given that option. So I placed the order to switch both of my phone lines over to fiber.

At some point I learned that each phone number at Frontier is on a separate account. This was completely invisible to me because both of my phone lines are billed on the account for my primary number, so that's the only account I see. Some of the Frontier people I talked to were able to find the separate account for the secondary line, but it always seemed to take them a while. In the end, I think that the fact that the secondary phone was actually a separate account has saved me some hassle with it: because it was on a separate account, the order to change the second phone over to fiber was done with a separate work order, scheduled for the day following the primary work order. Once the trouble started, I was able to cancel that second work order before anything was done to the second line; but the work had already started on the first line, and that has been the headache. I wonder now if there was any way I could have convinced them to just treat the internet service as a new internet-only account and so leave the phone lines and their account completely untouched.

I was pleased that my installation was scheduled very quickly, just two days later, on April 11. I should not have been. As it turned out, I did not actually get my FiOS service until April 18.

I wonder, had I known then what I know now, what I might have been able to do to avoid any of the troubles I have had.

Trouble

On the morning of April 11, I was a bit surprised that the installer did not call first to confirm I was home before coming by. When he arrived, I learned why: although he had two different phone numbers for me, somehow he had typos in both of them. These two numbers were for my two Frontier phone lines. I would have thought the computer would have just copied those numbers into the work order, but I assume now that a person manually put in those phone numbers, and somehow got them both wrong.

Unfortunately, the person who took my order scheduled my installer visit without first scheduling the preceding two steps of the installation process. As a result, when the installer came out for the April 11 appointment, he was unable to do his work, and had to leave having done nothing.

Before that first installer left, he told me he would call in the work orders to do the steps that should have been done before he got there. He might have done this right away, but when I called Frontier a little bit later that day, I was still unable to reschedule the installation because they didn't have the notes from that day's work order yet. So I had to wait and call back a couple of days later.

I was disappointed that I would have to wait longer to get my fast internet service, but that was just a mild disappointment. What was more annoying was that my DSL service went out on April 13, two days after that original installation date.

As I mentioned above, due to my unusual DSL situation, it was very difficult for me to get anyone to take any action on my DSL line. I called my ISP, and they said everything looked fine to them. I called Frontier and they couldn't help me at all; they had absolutely zero visibility into my DSL service. One tech said he would run a DSL line check on my line, but the computer wouldn't let him because it said there was no DSL service on my line.

My ISP suggested that my DSL modem may have died, and while I admit that is a possibility, the timing of the outage, plus the fact that the modem lights indicated no DSL carrier, leads me to believe that the work order to switch my copper line to fiber triggered some follow-on internal work order to turn off the DSL on that line, and because my DSL service was invisible to everyone who looked at my account, they had no way to manage that internal work order.

After a few frustrating and fruitless phone calls trying to get my DSL line fixed, I decided to forget it and hope that my new fiber internet connection would be running soon. In the meantime, I tethered my computer to my phone when I wanted to use the internet, so I did not have to suffer internet withdrawal while waiting for FiOS. Ironically, this gave me a faster connection than my 3Mbps DSL line, although I never got it working as a gateway for my entire LAN, but only used it on one computer at a time.

The first step in the installation process is for the utility locators to come out and spray lines marking the location of the existing utilities so that the people burying the fiber don't damage any existing buried utilities. Two days after the aborted initial installation appointment, on the same day my DSL service went out, various colored lines started appearing in my front yard marking the utilities. The following morning the fiber installers came out and buried the fiber cable running from the curb to my house (yay!). BUT - that afternoon, yet another utility locator came out to locate more utilities. So the fiber installers jumped the gun by installing the fiber before all of the utilities were located. Fortunately, they did not damage any of the unlocated utilities, so although they did not follow the prescribed procedure, at least no harm resulted from that mistake.

On April 18, now that the fiber was in place, the second installer came out to finish the installation. In about two hours he installed all the equipment and got the FiOS internet service working (yay!). For much of the next hour he worked over the phone with a technician trying to get the primary phone line working over fiber. After some discussion with me, they finally gave up and moved the phone line back to copper.

I was perfectly happy keeping my phone service on copper, as that's what I had originally wanted anyway. If only it had been so easy.

I learned from the installer that the second phone line was on a separate work order, to be moved from copper to fiber the next day. Given that they were unable to move the first line, and were willing to keep it on copper (I thought), I called and canceled the service call that was scheduled for the next day. I'm pretty sure doing that has saved me a lot of grief on my second phone line, as so far I have not had any problems with it, and it has continued working just fine on copper, as well as being billed properly.

On April 25, one week after the FiOS installation, I learned that my primary phone was not working properly. It may be that it stopped working a day or two sooner, but this is the day I realized it. It was broken in a strange way: I could place outgoing calls, and I could receive incoming calls from another phone number in the same exchange, such as my second phone line, but calls from outside the exchange would not go through. When I called from my mobile phone, which has a different area code, I could hear a ringback on my mobile, but my landline never rang. When I called from my wife's mobile phone, which is in the same area code but not in the same exchange, I immediately got a message saying "Your call can not be completed." I spent a couple of hours on the phone with Frontier over this.

On April 30, five days later, they finally managed to get the phone working again. We got a call at 8:15am that Sunday morning from a repair man testing to see if the line was working. Fortunately, we were already awake.

Two days later, on May 2, the phone service went out again, in the same way. Another hour on the phone with Frontier, and this time it "only" took them two days to get it fixed. So far, from then until now (mid-June), the phone service has not gone out again, so I am hopeful that they really have fixed it.

On May 8, I received my first bill from Frontier since getting my new FiOS service. It had a couple of minor errors on it, which I was able to deal with on the phone to Frontier in about 15 minutes.

On June 7, I received my second bill from Frontier since getting my new FiOS service. This one had more serious problems, and I spent closer to an hour on the phone with Frontier. The most significant problem is that, although my phone service never got switched over to fiber, which also would have included switching to a new service plan, the billing did get switched to the new plan. My old plan was $18.90/month, the new plan is $30.99/month. So I am being charged an extra $12.09 for exactly the same service that I was getting before the FiOS installation. The billing person I talked to told me she was unable to change my phone service back to the old plan because I had been grandfathered in at that old rate. I assume the computer did not provide her any way to go back to that grandfathered rate.

So here I am, two months after ordering FiOS, trying to figure out what I should do about my phone service. Try harder to get it back to the old rate? Try to get it changed to the service plan I am now being forced to pay for?

Or maybe I should just cancel it. Who has land lines these days anyway?

Mistakes

Here is a list of what I believe are the mistakes Frontier made that led to the above trouble.
  • When taking my order, the service rep scheduled the equipment installation without first scheduling utility location and fiber installation
  • Both of my phone numbers were entered incorrectly in the original work order
  • The fiber installers buried the fiber before all of the utilities were located
  • When the original installation was postponed, the order to disconnect my DSL service was not also postponed
  • When the installer was unable to move the phone service to fiber, and kept it on copper, he should have canceled the rest of the service order for moving the phone service (although I suspect the computer would not have let him do that, since he had already done some of the work on it)
  • When the phone went out the first time, and the repair man got it working again, he must have missed some piece of the puzzle, since it went out again two days later
  • Given that the phone service never actually got switched to the new plan on fiber, the billing likewise should not have changed

Good Stuff

While I think far more has gone badly than is reasonable, not everything has gone wrong. In fairness, I list here some good things.
  • The fiber installer did a very nice job burying the fiber line from the curb to the house. We could hardly see where they ran it, including through sod, and even where they had to run it under a bed of solid pachysandra, they only damaged a strip a few inches wide.
  • The equipment installer cheerfully ran ethernet cable from the ONT, across the ceiling in my garage, through a wall, into my network equipment closet, and to a wall-mounted jack.
  • My 100/100 internet service came up smoothly on the (second) scheduled date, and has been working well ever since. It is satisfyingly fast.
  • When I run speed tests, I consistently do get 100Mbps both up and down.
  • The Arris wifi router they included in the installation was actually pretty nice (although it would be better if there were some documentation available somewhere). If I were a less technically demanding customer, I would probably still be using it.
  • Both installers who came to my house were friendly and competent. A few of the tech support people I talked to also seemed quite competent.
  • Almost everyone I have communicated with at Frontier has been friendly and has (as far as I can tell) tried their best to help me. They always let me stay on the line asking questions as long as I wanted to; I never felt anyone was trying to get me to hang up.
  • I have not had any trouble getting credits applied to my bill.

Answers

This section lists what I think are the answers to the four questions I started with. YMMV: service, equipment, processes and prices may vary across regions and over time, and depending on your situation.

What service options do I have?

Sadly, I can't give you good answers here, so you will probably have to call or chat with Frontier and experience your own frustration at getting a different answer each time.

I do, however, have a few things to point out.

One point, that was always unclear to me when researching FiOS, is that there is no technical reason you can not keep your copper-wire phone along with FiOS. The fiber line is installed completely independently of the copper wires, and the service is likewise independent. Frontier may tell you that you must switch your phone service over to fiber service (either TDM, in which the phone signal is sent over the fiber separately from the Internet signal, or VoIP, where it is sent on top of the Internet signal), but that is purely a business issue.

A possible sticking point is the way Frontier handles their accounts: if your phone service is on the same account as your FiOS internet service, they are constrained as to what the computer will let them do with that phone service. If you want to keep your copper phone lines and they are telling you you can't, perhaps you can ask to put the Internet service on a separate account. You can then ask to have both accounts billed together. But you might lose out on some bundling discounts this way.

One of the differences between POTS over copper wires and VoIP is that POTS is regulated phone service, but VoIP is not. More specifically, under the Telecommunications Act of 1996 VoIP is considered an information service rather than a communications service, the upshot being that you don't have the same level of guarantees as POTS, which is regulated as a communications service. However, IANAL, and I was unable to determine whether or how later laws may have modified this situation, or whether those regulations are still being enforced, so this may be a moot point.

What equipment will be installed and where?

Not including the fiber from the street to your house that gets buried as part of the installation process, the installer installs three pieces of equipment:
  1. The ONT (Optical Network Terminal), which converts between the optical signal carried on the fiber to the electrical signals used in the house. The ONT has the following connections:
    • An optical connection that gets connected to the fiber from the street
    • Two 8P8C (RJ45) ethernet jacks for the internet connection
    • Two RJ-11 jacks for phone connections
    • A coaxial connector for the cable connection
    The ONT can be configured to provide internet service either through the 8P8C connector on a standard ethernet cable, or through the coaxial cable using MOCA.
    The ONT is typically mounted on the outside of the garage. The fiber from the street is routed first into a holding box, typically mounted behind the actual ONT, where the excess cable is wrapped in big loops to take up all the slack, then from there it enters the ONT.
  2. A power supply that includes a small battery backup for the ONT. This is typically mounted inside the garage, ideally just opposite where the ONT is mounted on the outside, and near a power outlet. The installer will then drill a hole through the garage wall to feed through the power wire from the supply to the ONT, and possibly another to bring the ethernet and coaxial cables into the garage if they will be routed through the garage. By default, the battery backup provides power only for the phone lines. It can be hacked to provide power for the internet portion of the ONT, or you can just buy your own UPS and plug the ONT power supply into that (although Frontier recommends plugging the ONT power supply directly into an outlet).
  3. A MOCA-capable router. In my case this was an Arris NVG468MQ, which is a reasonably nice wireless router, except that they didn't give me a manual, and I was unable to find anything of substance online. The router has the following connections:
    • A WAN ethernet port
    • Four LAN ethernet ports
    • A coax connector in case the internet signal is being supplied using MOCA
    • A four-wire RJ-11 phone jack for up to two phone lines
    If you have a good installer, they should be willing to let you decide where you want to put your router, and run ethernet cable (or coax if using MOCA) to that location, including drilling holes and installing a wall jack.
The internet signal from the ONT to the router can run either over an ethernet cable or over a coax cable. If you are getting TV service, they will have to run a coax cable for the TV service. If your internet service is slower than 100/100, it is possible to run the internet service over that same cable to the MOCA-capable router. If your internet service is 100/100 or faster, you probably want to run that over an ethernet cable; and you might someday want to upgrade to 100/100 or faster service later, so you probably should have them install that ethernet cable now anyway and have them run the internet signal through that to the router. Plus, that gives you the option of replacing their router with one of your own choice that doesn't do MOCA.

What is the installation process?

Installation of new FiOS service - not including preliminary research, placing the order, and post-installation followup to correct problems - consists of three sequential steps:
  1. Locate existing utilities: one or more people come out with metal detectors that they use to locate existing utilities such as power, water, sewer, gas, phone, and cable, and paint different colored lines marking those locations so that the fiber installers don't accidentally damage the existing utilities.
  2. Bury fiber from curb to house: a fiber installer puts in that last piece of fiber from the drop point (by the street near your house) to your house, typically to the garage. In the other direction, the fiber at the curb runs to a nearby junction box, where the installer connects it to an available port. At this point a signal is available at the fiber end by the house.
  3. Install equipment outside and inside the house: an equipment installer installs the equipment on the outside of your house and inside your house, and connects everything up. If you have existing POTS service and are switching to FiOS phone service, the phone lines that lead into the house are disconnected from the old copper lines and connected to the output of the ONT. The installer calls the plant and works with them to bring up the services you have ordered.

How much will it cost?

Perhaps because I am a long-time customer, Frontier did not charge me any kind of installation fee, which was nice. I don't know if that is standard. One person told me the regular installation fee is $80.

For the monthly fees, it may cost significantly more than you expect.

Frontier advertises their 100/100 internet service as $60 per month. They have not yet managed to send me a clean monthly bill since my upgrade, but based on my estimate of what that monthly amount is going to be, I believe the effective cost of my 100/100 service is actually over $100 per month. Here's how that breaks down:
  • The $60 rate is only if you sign a two year contract and only for the first six months. This is stated in the fine print on their web page, along with "Equip. and other fees apply." I did not sign a contract, so my monthly fee is $85.
  • After Frontier told me I was required to change my phone service to a new plan, and then was unable to deliver, my old grandfathered-in rate of $18.90 disappeared and was replaced by the $30.99 rate for Digital Phone Unlimited, despite the fact that I don't actually have that service. So I am currently paying an additional $12.09 per month for exactly the same phone service that I had before ordering FiOS internet service.
  • Taxes look like they will be about an additional $6 per month.
One other annoyance relating to cost: Frontier offered me a $100 gift card for signing up with them for FiOS internet. When I went to activate the gift card on their web site, I was presented with a terms and conditions screen requiring me to agree to a new 1 year term agreement. I had chosen not to sign a contract and to pay $85/month rather than $60/month, so it felt kind of like they were trying to pull a fast one on me by hoping I would activate the gift card without reading the fine print.

Frontier's Problems

  • Frontier's web site does not provide very good information about what service options are available.
  • If you call their Customer Service outside of their working hours, you get a message telling you they are closed, but that message does not tell you when they are open, and it's not an easy thing to find on their web site.
  • Different people at Frontier will give you different answers to the same questions. For example, I asked whether I would need to upgrade my copper-wire phone service to fiber; some said yes, some said no. Or sometimes first one answer then the other. One person suggested I put my phone service on a separate account from my internet service; another told me I could not do that.
  • Frontier's phone bills provide tons of details about taxes, but almost no details about regular charges. For example, I have two phone lines, and for most of the last few years they were billed as one line item labeled "Residence Line", with no indication that there were two lines.
  • Frontier's computers significantly constrain what their people can see and do. Or maybe their programs are just really hard to use. The customer service reps can't see the details of service calls, and the service techs can't see the account details. It is apparently not obvious when a customer has multiple accounts being billed together. And nobody could see anything about my DSL line.

Timeline

Date Event
2017-03-02 Th Online chat #1 with Frontier (43 minutes)
2017-03-06 Mo Online chat #2 with Frontier (23 minutes)
2017-03-15 We Online chat #3 with Frontier (55 minutes)
2017-03-18 Sa Online chat #4 with Frontier (20 minutes, then cut off)
2017-03-20 Mo Online chat #5 with Frontier (estimated 20 minutes)
2017-03-21 Tu Online chat #6 with Frontier (1 hour and 38 minutes)
2017-04-09 Su Phone call with Frontier to order FiOS, service scheduled for Apr 11 (1 hour and 17 minutes)
2017-04-11 Tu Installer came out, couldn't do anything because they have not yet buried the fiber from the curb to the house
2017-04-11 Tu Called Frontier to reschedule installation, was told the current installer has not yet entered his notes, please call back in 24 hours (12 minutes)
2017-04-13 Th DSL service died at about 12:30pm
2017-04-13 Th Utility locators started painting colored lines where existing services are buried
2017-04-13 Th Called Frontier to try to get DSL line fixed (24 minutes)
2017-04-14 Fr Fiber installers installed the curb-to-house fiber (before all the locators had painted their lines)
2017-04-14 Fr Another locator came out to paint lines; when I pointed out that the fiber had already been installed, he stopped painting, took his final photos, and left
2017-04-14 Fr Called ISP to try to get DSL line fixed (12 minutes)
2017-04-14 Fr Called Frontier (multiple times) to check on status of FiOS order (the fiber was installed this morning, but they said the order had not yet been updated to show that) (8 minutes + 13 minutes + 12 minutes + 25 minutes)
2017-04-15 Sa Called Frontier to check on the status of my FiOS order (8 minutes)
2017-04-18 Tu Installer came out and completed the physical installation of the equipment, got the FiOS internet service working. He was unable to get the phones working over fiber, so switched everything back to copper and left, with everything working (3 hours and 10 minutes)
2017-04-18 Tu Called Frontier, canceled the remaining order to move the second line over to fiber (scheduled for tomorrow) (7 minutes)
2017-04-25 Tu Our main line stopped working, was unable to be reached from outside our exchange
2017-04-25 Tu Called Frontier to report our main phone line not working (44 minutes)
2017-04-26 We Called Frontier to continue discussions about non-working phone (1 hour and 35 minutes)
2017-04-30 Su Received a call from Frontier at about 8:15am this morning on the main line, he said it was now fixed (1 minute)
2017-05-01 Mo Phone seems to have been working today, we received at least one incoming phone call
2017-05-02 Tu Called Frontier in the morning because my main phone was not working again (39 minutes, then was cut off)
2017-05-02 Tu My wife called Frontier mid-day about the non-working phone (15 minutes)
2017-05-02 Tu Called Frontier in the evening to continue the call from this morning (14 minutes)
2017-05-04 Th Frontier called, the line is working again
2017-05-08 Mo Called Frontier to have them correct errors on my April bill (the first received since I started FiOS service) (12 minutes)
2017-06-07 We Received second bill since switching to FiOS - still wrong
2017-06-14 We Called Frontier to deal with problems on my May bill (48 minutes)

Total time (as of June 14): 20.3 hours
  • Web research: 5 hours
  • Chat: 4.3 hours
  • Place order: 1.3 hours
  • Installer: 3.2 hours
  • Followup phone calls (through June 14): 6.5 hours

Selected Quotes

I took notes on all my phone calls with Frontier, including writing down certain things verbatim. For your entertainment, I present here some of those quotes, in no particular order. I will let you imagine the context.
  • That is very confusing.
  • Why can't I see that one?
  • I don't know why they didn't just leave it alone.
  • The program is wrong.
  • How are you an R-U out of Washington?
  • ... and that's what I'm not seeing.
  • We don't do these very often.
  • Within our system we have nine different portals where we have to test things.
  • This is very new to me, I have never dealt with two lines like this.
  • Sorry this is taking so long, we'll get it figured out for you.
  • It's not giving me anything.