Thursday, June 22, 2017

FiOS - A Cautionary Tale

I delayed signing up for Frontier FiOS because I was concerned they might screw things up. I should have been more concerned.

This is a long post. Consider it entertainment. Or just skip to the Answers.

Contents

The Need for Speed

I have had internet connectivity for decades, starting back with modems so slow, I knew people who had to pause in their typing to let the modem catch up. I appreciated every doubling of speed as each generation of modem arrived. I was surprised when modem speeds reached 4800 and then 9600 baud - how could you get more bits per second than the 3K bandwidth of a phone line? - and I was astounded with the jump to 56K modems.

When DSL came out, I waited impatiently for it to be available in my neighborhood, and signed up as soon as I could. After years of using a 56K modem, my 740Kbps DSL line was satisfyingly fast.

I lived with 740Kbps for six years, until one day my DSL modem broke. While researching new modems, I learned that I could have my service switched from Frame Relay to ATM and bump up my speed to 3Mbps. Normally this would mean my service would be out for 10 days while they did that, but since it was already out, it seemed like a good time to make the switch. The speed bump from 740Kbps to 3Mbps was a mere 4x, far less than the 13x increase from the 56K modem to 740Kbps DSL, but still, 3Mbps was satisfyingly fast.

Verizon actually offered FiOS in my neighborhood fairly early, but I was pretty happy with my DSL service, and I wasn't doing anything that I thought needed more than 3Mbps bandwidth. I remained happy with my 3Mbps for over a decade. But technology marches on; I bought some HDTVs, started watching more YouTube, and started working from home more often using bandwidth-hungry remote desktop applications. My 3Mbps connection was not sufficient to stream HDTV movies and YouTube clips, and my remote desktop experience was annoyingly slow. I was finally feeling the bandwidth squeeze.

Still, I delayed upgrading to FiOS. I had heard that I would have to give up my copper-wire land lines, which I was not keen to do. Some years ago our power was out for over a week; batteries everywhere ran down - even the local cell towers ran out of juice after a few days, so there was no cell service in our neighborhood - but, with our copper wires, we had phone service the whole time. I liked that.

In addition, by this time Verizon had sold to Frontier, and based on my experience and anecdotes I read, I was concerned that Frontier would mess something up when dealing with my service change request, particularly since my situation was rather unusual.

My Unusual Situation

In my case, there were a number of things about my situation that gave me pause when thinking about asking for any kind of change.
  1. I have a land line. This has become increasingly rare, and it seems Frontier is deprioritizing phone service so they can focus on providing internet and television service. It seems they want to provide packages that include everything, or at least include both internet and television.
  2. Actually, I have two land lines. I'm not sure I know anybody else who has two land lines at home any more. I used to have three, but finally got rid of the third line after disposing of my last FAX machine years ago. Although I have two lines, they have not both shown up on my monthly bill for many years now. Oh, I am paying for two lines, it's just that the second line is not itemized anywhere. If you didn't know a-priori that I had two phone lines, you would hardly be able to tell that by looking at my phone bill. Based on conversations I have had with support and billing people at Frontier, it's not obvious to them either, although after I point it out, and with enough digging, some of them could figure it out.
  3. I had a DSL line. As I mentioned, I delayed for quite some time in switching from DSL to FiOS. The longer I delayed, the fewer people had DSL lines, and the less Frontier cared about them. For this particular problem, I suppose my delaying upgrading perhaps made things worse.
  4. My DSL service provider was not Frontier. This caused a fair amount of frustration any time I had a service issue with my DSL line.
My DSL service was perhaps the most unusual part of my situation. Back in 1999, when I originally ordered DSL, GTE (yes, it was that long ago!) had partnerships with Internet Service Providers who provided the actual internet service. These are known as CLECs (Competitive Local Exchange Carriers). So GTE provided the line, and my selected ISP provided the internet service. Originally I paid GTE directly for the line and I paid the ISP for the internet service. But when I switched from Frame Relay to ATM service, my billing also changed so that I paid everything to the CLEC, and they paid Verizon for the line.

Back then many of the carrier ISPs had annoying policies such as blocking some ports, so it was nice to be a customer of a smaller ISP that was more interested in making its customers happy. The downside was that, whenever there was a service problem, I had to deal with two companies, and they each tended to say it was the other company's problem.

By the time I was considering switching from DSL to FiOS this year, it had become perhaps comically bad: when I talked to support and billing people at Frontier, they were completely unaware that I had DSL service on my Frontier phone line, and even with a lot of digging, nobody I talked to at Frontier this year was ever able to find even a trace of information about my DSL line.

On the other side, my ISP had been acquired multiple times over the years, each time by a larger and more remote company, until by this year they were no longer in the DSL business and no longer in the residential ISP business. Somehow through all this, my residential DSL line kept working, but I did start to feel I was skating on ever-thinning ice.

It was time to take the plunge and upgrade.

Questions

Before ordering FiOS service, I wanted to get the answers to four questions:
  1. What service options do I have?
  2. What equipment will be installed and where?
  3. What is the installation process?
  4. How much will it cost?
How does one answer questions like these in today's world? Hit up the internet, of course.

Research

Frontier's Web Site

I started by browsing Frontier's web site looking for information about their service offerings.

NOTE: Frontier's offerings are regional, so you may see different web pages than what I describe.

Their FiOS page shows four levels of service: 50Mbps, 75Mbps, 100Mbps or 150Mbps. Since I wanted both internet and phone service, I headed over to the bundles page to see what I could get. I don't want their television service, so I unchecked the "Video" box. This shows three bundles: two that include 30Mbps internet service (30? that's not one of the speeds listed on their FiOS page!) and one that includes 50Mbps. Do they offer bundles that include internet service faster than 50Mbps? Their web site doesn't say.

Their Phone page shows me information about copper-line phone service (lower in the page it says "Our reliable copper power stays on even when the power goes out or in an emergency"), where they list two plans that differ by $3. Confusingly, this phone service - you know, POTS using analog signals on copper wires - is called "Digital Essentials." Are there any other optional add-ons? There are a fair number of features bundled with the basic phone service, and a lot more bundled with that extra $3, but is that it? Like, I currently have an unlisted phone number, what's the charge for that? Sorry, that kind of stuff is not on their web site. Ah, here on the the Digital Phone Unlimited page, it says "Optional international calling packages are available for great savings", so apparently there are other options available - but it's not a link, so I have no idea what kind of packages they might offer.

How about a second phone line, how much does that cost? Sorry, that's not on the web page. VoIP? Oh, maybe you mean FiOS Digital Voice. Beats me what the scoop is on that. If you go to Frontier's FiOS Bundles page, where it says the phone service in their bundles is Digital Phone Unlimited, and you click on the Learn More button for the phone service, it takes you to that phone page I mentioned above - you know, the one that says "Our reliable copper power stays on even when the power goes out or in an emergency." So, if I get a bundle that includes FiOS internet, does that bundle include Digital Phone Unlimited running on copper wires?

The details of the above web pages are what they look like now, in June 2017. I believe they have changed since I did my initial research a few months ago, but the gist is the same: I was unable to figure out what options were available to me by reading their web site.

Besides looking at Frontier's web pages, I did a lot of Googling and browsing of other web sites. I learned a lot in general about equipment, but it was hard to know how much of it would apply to my situation. Although I did not record the time I spent browsing Frontier's and others' web sites, I estimate it was probably about five hours.

It was time to move on to online chat to get more answers.

Online Chat

I had six online chats with Frontier, totaling about 4 1/2 hours. Between each chat I did more online research, looking for details about the equipment and the installation experience both inside and outside the house. It was difficult to get a good handle on these details, particularly since I sometimes got conflicting answers from the Frontier people I chatted with.

For example, one of my questions was whether I could keep my copper phone lines, or whether I would be required to switch my phone service to fiber. One of the people I chatted with said this:
You would have to switch to a digital phone service ! Voip. Which basically means Voice over Internet
Another one said I could keep my phone service on copper and get their "Simply FiOS" service, which is fiber with only internet service.

Ordering

Once I reached the point where I felt I had as good answers as I could get - which admittedly were not always very good - it was time to place my order.

On April 9 I called Frontier to place my order. I would say the fact that it took me well over an hour to place my order was the first hint of trouble, but in truth there were plenty of hints during the many chats I had, where I was not getting consistent answers.

Part of the reason the phone call took so long was due to my unusual situation. The DSL was not much of an issue during ordering, since it was completely invisible to them and they couldn't do anything about it. The real trouble was that second phone line. Figuring out how to deal with that took probably 45 minutes.

When I asked if I was required to switch my phone service from copper to fiber, the service rep first said no, but then went and asked someone else, came back and said yes, I would have to switch. I would have preferred to keep my phones on copper (and especially I would have preferred it given how much trouble I have had with the switch), but I was not given that option. So I placed the order to switch both of my phone lines over to fiber.

At some point I learned that each phone number at Frontier is on a separate account. This was completely invisible to me because both of my phone lines are billed on the account for my primary number, so that's the only account I see. Some of the Frontier people I talked to were able to find the separate account for the secondary line, but it always seemed to take them a while. In the end, I think that the fact that the secondary phone was actually a separate account has saved me some hassle with it: because it was on a separate account, the order to change the second phone over to fiber was done with a separate work order, scheduled for the day following the primary work order. Once the trouble started, I was able to cancel that second work order before anything was done to the second line; but the work had already started on the first line, and that has been the headache. I wonder now if there was any way I could have convinced them to just treat the internet service as a new internet-only account and so leave the phone lines and their account completely untouched.

I was pleased that my installation was scheduled very quickly, just two days later, on April 11. I should not have been. As it turned out, I did not actually get my FiOS service until April 18.

I wonder, had I known then what I know now, what I might have been able to do to avoid any of the troubles I have had.

Trouble

On the morning of April 11, I was a bit surprised that the installer did not call first to confirm I was home before coming by. When he arrived, I learned why: although he had two different phone numbers for me, somehow he had typos in both of them. These two numbers were for my two Frontier phone lines. I would have thought the computer would have just copied those numbers into the work order, but I assume now that a person manually put in those phone numbers, and somehow got them both wrong.

Unfortunately, the person who took my order scheduled my installer visit without first scheduling the preceding two steps of the installation process. As a result, when the installer came out for the April 11 appointment, he was unable to do his work, and had to leave having done nothing.

Before that first installer left, he told me he would call in the work orders to do the steps that should have been done before he got there. He might have done this right away, but when I called Frontier a little bit later that day, I was still unable to reschedule the installation because they didn't have the notes from that day's work order yet. So I had to wait and call back a couple of days later.

I was disappointed that I would have to wait longer to get my fast internet service, but that was just a mild disappointment. What was more annoying was that my DSL service went out on April 13, two days after that original installation date.

As I mentioned above, due to my unusual DSL situation, it was very difficult for me to get anyone to take any action on my DSL line. I called my ISP, and they said everything looked fine to them. I called Frontier and they couldn't help me at all; they had absolutely zero visibility into my DSL service. One tech said he would run a DSL line check on my line, but the computer wouldn't let him because it said there was no DSL service on my line.

My ISP suggested that my DSL modem may have died, and while I admit that is a possibility, the timing of the outage, plus the fact that the modem lights indicated no DSL carrier, leads me to believe that the work order to switch my copper line to fiber triggered some follow-on internal work order to turn off the DSL on that line, and because my DSL service was invisible to everyone who looked at my account, they had no way to manage that internal work order.

After a few frustrating and fruitless phone calls trying to get my DSL line fixed, I decided to forget it and hope that my new fiber internet connection would be running soon. In the meantime, I tethered my computer to my phone when I wanted to use the internet, so I did not have to suffer internet withdrawal while waiting for FiOS. Ironically, this gave me a faster connection than my 3Mbps DSL line, although I never got it working as a gateway for my entire LAN, but only used it on one computer at a time.

The first step in the installation process is for the utility locators to come out and spray lines marking the location of the existing utilities so that the people burying the fiber don't damage any existing buried utilities. Two days after the aborted initial installation appointment, on the same day my DSL service went out, various colored lines started appearing in my front yard marking the utilities. The following morning the fiber installers came out and buried the fiber cable running from the curb to my house (yay!). BUT - that afternoon, yet another utility locator came out to locate more utilities. So the fiber installers jumped the gun by installing the fiber before all of the utilities were located. Fortunately, they did not damage any of the unlocated utilities, so although they did not follow the prescribed procedure, at least no harm resulted from that mistake.

On April 18, now that the fiber was in place, the second installer came out to finish the installation. In about two hours he installed all the equipment and got the FiOS internet service working (yay!). For much of the next hour he worked over the phone with a technician trying to get the primary phone line working over fiber. After some discussion with me, they finally gave up and moved the phone line back to copper.

I was perfectly happy keeping my phone service on copper, as that's what I had originally wanted anyway. If only it had been so easy.

I learned from the installer that the second phone line was on a separate work order, to be moved from copper to fiber the next day. Given that they were unable to move the first line, and were willing to keep it on copper (I thought), I called and canceled the service call that was scheduled for the next day. I'm pretty sure doing that has saved me a lot of grief on my second phone line, as so far I have not had any problems with it, and it has continued working just fine on copper, as well as being billed properly.

On April 25, one week after the FiOS installation, I learned that my primary phone was not working properly. It may be that it stopped working a day or two sooner, but this is the day I realized it. It was broken in a strange way: I could place outgoing calls, and I could receive incoming calls from another phone number in the same exchange, such as my second phone line, but calls from outside the exchange would not go through. When I called from my mobile phone, which has a different area code, I could hear a ringback on my mobile, but my landline never rang. When I called from my wife's mobile phone, which is in the same area code but not in the same exchange, I immediately got a message saying "Your call can not be completed." I spent a couple of hours on the phone with Frontier over this.

On April 30, five days later, they finally managed to get the phone working again. We got a call at 8:15am that Sunday morning from a repair man testing to see if the line was working. Fortunately, we were already awake.

Two days later, on May 2, the phone service went out again, in the same way. Another hour on the phone with Frontier, and this time it "only" took them two days to get it fixed. So far, from then until now (mid-June), the phone service has not gone out again, so I am hopeful that they really have fixed it.

On May 8, I received my first bill from Frontier since getting my new FiOS service. It had a couple of minor errors on it, which I was able to deal with on the phone to Frontier in about 15 minutes.

On June 7, I received my second bill from Frontier since getting my new FiOS service. This one had more serious problems, and I spent closer to an hour on the phone with Frontier. The most significant problem is that, although my phone service never got switched over to fiber, which also would have included switching to a new service plan, the billing did get switched to the new plan. My old plan was $18.90/month, the new plan is $30.99/month. So I am being charged an extra $12.09 for exactly the same service that I was getting before the FiOS installation. The billing person I talked to told me she was unable to change my phone service back to the old plan because I had been grandfathered in at that old rate. I assume the computer did not provide her any way to go back to that grandfathered rate.

So here I am, two months after ordering FiOS, trying to figure out what I should do about my phone service. Try harder to get it back to the old rate? Try to get it changed to the service plan I am now being forced to pay for?

Or maybe I should just cancel it. Who has land lines these days anyway?

Mistakes

Here is a list of what I believe are the mistakes Frontier made that led to the above trouble.
  • When taking my order, the service rep scheduled the equipment installation without first scheduling utility location and fiber installation
  • Both of my phone numbers were entered incorrectly in the original work order
  • The fiber installers buried the fiber before all of the utilities were located
  • When the original installation was postponed, the order to disconnect my DSL service was not also postponed
  • When the installer was unable to move the phone service to fiber, and kept it on copper, he should have canceled the rest of the service order for moving the phone service (although I suspect the computer would not have let him do that, since he had already done some of the work on it)
  • When the phone went out the first time, and the repair man got it working again, he must have missed some piece of the puzzle, since it went out again two days later
  • Given that the phone service never actually got switched to the new plan on fiber, the billing likewise should not have changed

Good Stuff

While I think far more has gone badly than is reasonable, not everything has gone wrong. In fairness, I list here some good things.
  • The fiber installer did a very nice job burying the fiber line from the curb to the house. We could hardly see where they ran it, including through sod, and even where they had to run it under a bed of solid pachysandra, they only damaged a strip a few inches wide.
  • The equipment installer cheerfully ran ethernet cable from the ONT, across the ceiling in my garage, through a wall, into my network equipment closet, and to a wall-mounted jack.
  • My 100/100 internet service came up smoothly on the (second) scheduled date, and has been working well ever since. It is satisfyingly fast.
  • When I run speed tests, I consistently do get 100Mbps both up and down.
  • The Arris wifi router they included in the installation was actually pretty nice (although it would be better if there were some documentation available somewhere). If I were a less technically demanding customer, I would probably still be using it.
  • Both installers who came to my house were friendly and competent. A few of the tech support people I talked to also seemed quite competent.
  • Almost everyone I have communicated with at Frontier has been friendly and has (as far as I can tell) tried their best to help me. They always let me stay on the line asking questions as long as I wanted to; I never felt anyone was trying to get me to hang up.
  • I have not had any trouble getting credits applied to my bill.

Answers

This section lists what I think are the answers to the four questions I started with. YMMV: service, equipment, processes and prices may vary across regions and over time, and depending on your situation.

What service options do I have?

Sadly, I can't give you good answers here, so you will probably have to call or chat with Frontier and experience your own frustration at getting a different answer each time.

I do, however, have a few things to point out.

One point, that was always unclear to me when researching FiOS, is that there is no technical reason you can not keep your copper-wire phone along with FiOS. The fiber line is installed completely independently of the copper wires, and the service is likewise independent. Frontier may tell you that you must switch your phone service over to fiber service (either TDM, in which the phone signal is sent over the fiber separately from the Internet signal, or VoIP, where it is sent on top of the Internet signal), but that is purely a business issue.

A possible sticking point is the way Frontier handles their accounts: if your phone service is on the same account as your FiOS internet service, they are constrained as to what the computer will let them do with that phone service. If you want to keep your copper phone lines and they are telling you you can't, perhaps you can ask to put the Internet service on a separate account. You can then ask to have both accounts billed together. But you might lose out on some bundling discounts this way.

One of the differences between POTS over copper wires and VoIP is that POTS is regulated phone service, but VoIP is not. More specifically, under the Telecommunications Act of 1996 VoIP is considered an information service rather than a communications service, the upshot being that you don't have the same level of guarantees as POTS, which is regulated as a communications service. However, IANAL, and I was unable to determine whether or how later laws may have modified this situation, or whether those regulations are still being enforced, so this may be a moot point.

What equipment will be installed and where?

Not including the fiber from the street to your house that gets buried as part of the installation process, the installer installs three pieces of equipment:
  1. The ONT (Optical Network Terminal), which converts between the optical signal carried on the fiber to the electrical signals used in the house. The ONT has the following connections:
    • An optical connection that gets connected to the fiber from the street
    • Two 8P8C (RJ45) ethernet jacks for the internet connection
    • Two RJ-11 jacks for phone connections
    • A coaxial connector for the cable connection
    The ONT can be configured to provide internet service either through the 8P8C connector on a standard ethernet cable, or through the coaxial cable using MOCA.
    The ONT is typically mounted on the outside of the garage. The fiber from the street is routed first into a holding box, typically mounted behind the actual ONT, where the excess cable is wrapped in big loops to take up all the slack, then from there it enters the ONT.
  2. A power supply that includes a small battery backup for the ONT. This is typically mounted inside the garage, ideally just opposite where the ONT is mounted on the outside, and near a power outlet. The installer will then drill a hole through the garage wall to feed through the power wire from the supply to the ONT, and possibly another to bring the ethernet and coaxial cables into the garage if they will be routed through the garage. By default, the battery backup provides power only for the phone lines. It can be hacked to provide power for the internet portion of the ONT, or you can just buy your own UPS and plug the ONT power supply into that (although Frontier recommends plugging the ONT power supply directly into an outlet).
  3. A MOCA-capable router. In my case this was an Arris NVG468MQ, which is a reasonably nice wireless router, except that they didn't give me a manual, and I was unable to find anything of substance online. The router has the following connections:
    • A WAN ethernet port
    • Four LAN ethernet ports
    • A coax connector in case the internet signal is being supplied using MOCA
    • A four-wire RJ-11 phone jack for up to two phone lines
    If you have a good installer, they should be willing to let you decide where you want to put your router, and run ethernet cable (or coax if using MOCA) to that location, including drilling holes and installing a wall jack.
The internet signal from the ONT to the router can run either over an ethernet cable or over a coax cable. If you are getting TV service, they will have to run a coax cable for the TV service. If your internet service is slower than 100/100, it is possible to run the internet service over that same cable to the MOCA-capable router. If your internet service is 100/100 or faster, you probably want to run that over an ethernet cable; and you might someday want to upgrade to 100/100 or faster service later, so you probably should have them install that ethernet cable now anyway and have them run the internet signal through that to the router. Plus, that gives you the option of replacing their router with one of your own choice that doesn't do MOCA.

What is the installation process?

Installation of new FiOS service - not including preliminary research, placing the order, and post-installation followup to correct problems - consists of three sequential steps:
  1. Locate existing utilities: one or more people come out with metal detectors that they use to locate existing utilities such as power, water, sewer, gas, phone, and cable, and paint different colored lines marking those locations so that the fiber installers don't accidentally damage the existing utilities.
  2. Bury fiber from curb to house: a fiber installer puts in that last piece of fiber from the drop point (by the street near your house) to your house, typically to the garage. In the other direction, the fiber at the curb runs to a nearby junction box, where the installer connects it to an available port. At this point a signal is available at the fiber end by the house.
  3. Install equipment outside and inside the house: an equipment installer installs the equipment on the outside of your house and inside your house, and connects everything up. If you have existing POTS service and are switching to FiOS phone service, the phone lines that lead into the house are disconnected from the old copper lines and connected to the output of the ONT. The installer calls the plant and works with them to bring up the services you have ordered.

How much will it cost?

Perhaps because I am a long-time customer, Frontier did not charge me any kind of installation fee, which was nice. I don't know if that is standard. One person told me the regular installation fee is $80.

For the monthly fees, it may cost significantly more than you expect.

Frontier advertises their 100/100 internet service as $60 per month. They have not yet managed to send me a clean monthly bill since my upgrade, but based on my estimate of what that monthly amount is going to be, I believe the effective cost of my 100/100 service is actually over $100 per month. Here's how that breaks down:
  • The $60 rate is only if you sign a two year contract and only for the first six months. This is stated in the fine print on their web page, along with "Equip. and other fees apply." I did not sign a contract, so my monthly fee is $85.
  • After Frontier told me I was required to change my phone service to a new plan, and then was unable to deliver, my old grandfathered-in rate of $18.90 disappeared and was replaced by the $30.99 rate for Digital Phone Unlimited, despite the fact that I don't actually have that service. So I am currently paying an additional $12.09 per month for exactly the same phone service that I had before ordering FiOS internet service.
  • Taxes look like they will be about an additional $6 per month.
One other annoyance relating to cost: Frontier offered me a $100 gift card for signing up with them for FiOS internet. When I went to activate the gift card on their web site, I was presented with a terms and conditions screen requiring me to agree to a new 1 year term agreement. I had chosen not to sign a contract and to pay $85/month rather than $60/month, so it felt kind of like they were trying to pull a fast one on me by hoping I would activate the gift card without reading the fine print.

Frontier's Problems

  • Frontier's web site does not provide very good information about what service options are available.
  • If you call their Customer Service outside of their working hours, you get a message telling you they are closed, but that message does not tell you when they are open, and it's not an easy thing to find on their web site.
  • Different people at Frontier will give you different answers to the same questions. For example, I asked whether I would need to upgrade my copper-wire phone service to fiber; some said yes, some said no. Or sometimes first one answer then the other. One person suggested I put my phone service on a separate account from my internet service; another told me I could not do that.
  • Frontier's phone bills provide tons of details about taxes, but almost no details about regular charges. For example, I have two phone lines, and for most of the last few years they were billed as one line item labeled "Residence Line", with no indication that there were two lines.
  • Frontier's computers significantly constrain what their people can see and do. Or maybe their programs are just really hard to use. The customer service reps can't see the details of service calls, and the service techs can't see the account details. It is apparently not obvious when a customer has multiple accounts being billed together. And nobody could see anything about my DSL line.

Timeline

Date Event
2017-03-02 Th Online chat #1 with Frontier (43 minutes)
2017-03-06 Mo Online chat #2 with Frontier (23 minutes)
2017-03-15 We Online chat #3 with Frontier (55 minutes)
2017-03-18 Sa Online chat #4 with Frontier (20 minutes, then cut off)
2017-03-20 Mo Online chat #5 with Frontier (estimated 20 minutes)
2017-03-21 Tu Online chat #6 with Frontier (1 hour and 38 minutes)
2017-04-09 Su Phone call with Frontier to order FiOS, service scheduled for Apr 11 (1 hour and 17 minutes)
2017-04-11 Tu Installer came out, couldn't do anything because they have not yet buried the fiber from the curb to the house
2017-04-11 Tu Called Frontier to reschedule installation, was told the current installer has not yet entered his notes, please call back in 24 hours (12 minutes)
2017-04-13 Th DSL service died at about 12:30pm
2017-04-13 Th Utility locators started painting colored lines where existing services are buried
2017-04-13 Th Called Frontier to try to get DSL line fixed (24 minutes)
2017-04-14 Fr Fiber installers installed the curb-to-house fiber (before all the locators had painted their lines)
2017-04-14 Fr Another locator came out to paint lines; when I pointed out that the fiber had already been installed, he stopped painting, took his final photos, and left
2017-04-14 Fr Called ISP to try to get DSL line fixed (12 minutes)
2017-04-14 Fr Called Frontier (multiple times) to check on status of FiOS order (the fiber was installed this morning, but they said the order had not yet been updated to show that) (8 minutes + 13 minutes + 12 minutes + 25 minutes)
2017-04-15 Sa Called Frontier to check on the status of my FiOS order (8 minutes)
2017-04-18 Tu Installer came out and completed the physical installation of the equipment, got the FiOS internet service working. He was unable to get the phones working over fiber, so switched everything back to copper and left, with everything working (3 hours and 10 minutes)
2017-04-18 Tu Called Frontier, canceled the remaining order to move the second line over to fiber (scheduled for tomorrow) (7 minutes)
2017-04-25 Tu Our main line stopped working, was unable to be reached from outside our exchange
2017-04-25 Tu Called Frontier to report our main phone line not working (44 minutes)
2017-04-26 We Called Frontier to continue discussions about non-working phone (1 hour and 35 minutes)
2017-04-30 Su Received a call from Frontier at about 8:15am this morning on the main line, he said it was now fixed (1 minute)
2017-05-01 Mo Phone seems to have been working today, we received at least one incoming phone call
2017-05-02 Tu Called Frontier in the morning because my main phone was not working again (39 minutes, then was cut off)
2017-05-02 Tu My wife called Frontier mid-day about the non-working phone (15 minutes)
2017-05-02 Tu Called Frontier in the evening to continue the call from this morning (14 minutes)
2017-05-04 Th Frontier called, the line is working again
2017-05-08 Mo Called Frontier to have them correct errors on my April bill (the first received since I started FiOS service) (12 minutes)
2017-06-07 We Received second bill since switching to FiOS - still wrong
2017-06-14 We Called Frontier to deal with problems on my May bill (48 minutes)

Total time (as of June 14): 20.3 hours
  • Web research: 5 hours
  • Chat: 4.3 hours
  • Place order: 1.3 hours
  • Installer: 3.2 hours
  • Followup phone calls (through June 14): 6.5 hours

Selected Quotes

I took notes on all my phone calls with Frontier, including writing down certain things verbatim. For your entertainment, I present here some of those quotes, in no particular order. I will let you imagine the context.
  • That is very confusing.
  • Why can't I see that one?
  • I don't know why they didn't just leave it alone.
  • The program is wrong.
  • How are you an R-U out of Washington?
  • ... and that's what I'm not seeing.
  • We don't do these very often.
  • Within our system we have nine different portals where we have to test things.
  • This is very new to me, I have never dealt with two lines like this.
  • Sorry this is taking so long, we'll get it figured out for you.
  • It's not giving me anything.

Sunday, December 21, 2014

The Rule Of Law

A layman's view of The Rule of Law. IANAL.

Contents

This I Believe

Some years ago NPR started running a series called This I Believe as a tribute to Edward R. Murrow and his original 1951 radio program of the same name. As I commuted I would occasionally catch an episode and hear an essay about the topic in which a contributor believed. I would listen to an essay around a weighty topic as God, Love, Funerals, Good and Evil or Public Service and think, "no", "maybe", or "yeah, sure". Then one day I heard Michael Mullane's essay on The Rule Of Law and I thought "Yes! This I believe!"

I particularly liked Michael's point that the Tinkerbell effect applies to the Rule of Law. As he says, God exists (or does not exist) whether or not you or I believe that to be so, but with the Rule of Law, it can only exist if almost all of us believe in it and follow it. As Death says to a human near the end of Terry Pratchett's Discworld book Hogfather, "YOU NEED TO BELIEVE IN THINGS THAT AREN'T TRUE. HOW ELSE CAN THEY BECOME?" (Death always talks IN CAPITAL LETTERS.)

The Importance of Law

Why is law important? The American Declaration of Independence asserts that "all men are created equal" and The Universal Declaration of Human Rights asserts that "all human beings are born free and equal in dignity and rights." To support that position we need a system of law that in fact treats all people equally. But even if the law does not protect all of the fundamental human rights, it can provide an important benefit to its society: stability through predictability.

To be predictable, the system of laws must be:
  • Understandable - the laws can be understood by most people.
  • Consistent - individual laws do not conflict with each other.
  • Extensive - the laws cover all common situations and a large portion of less common situations.
There have been many successful nations that followed the Rule of Law with different laws for different classes of people, including Rome and Greece. A system of law can provide stability and a foundation for an orderly and effective society without treating all people equally.

Prescriptive and Proscriptive Law

Prescriptive laws are those that tell us what we must do, such as Honour thy father and thy mother. Proscriptive laws are those that tell us what we must not do, such as "Thou shalt not kill".

You can think of prescriptive law as additive manufacturing: you can start with nothing, and add pieces until you get something useful, like building up a sculpture by adding little pieces of clay, or 3D printing.

Proscriptive law is more like subtractive manufacturing: you start with a block of something and carve away pieces until you get the desired result, like starting with a chunk of marble and carving a sculpture out of it, or machining.

(But don't try searching for additive law or subtractive law unless you are working with primary colors. :-) )

Given the assumption of freedom in both of the Declarations above, it's easier to start by saying people can do anything, then add proscriptive laws specifying what they can't do. Compared to the complete freedom and anarchy of a society with no laws, you can get pretty far down the road to stability just with proscriptive laws. Of the Ten Commandments, eight are proscriptive and only two are prescriptive.

Law versus Convention

While the Rule of Law normally refers to the explicit and codified laws on the books, which can be enforced by the state, there is another set of rules that most of us live by which are not legally mandated. These conventions include social guidelines that prescribe how to behave and communicate, including when and how it is appropriate to touch (such as shaking hands or a pat on the back), to ask for something (with "please" and "thank you"), to offer advice ("true/kind/necessary") or apologies, and many other behaviors.

These conventions don't have the force of law. If you break these rules, you won't be sent to jail or be forced to pay someone monetary damages - but you might find that you are a little less successful and your life might be a little less pleasant. Like laws, conventions are only useful if most of us agree on them, and like laws, a widely accepted and understood set of conventions helps make the world a little bit more predictable, which in turn makes it a little bit easier for people to make plans and be successful.

In effect, social conventions are simply another layer of "laws" that sit below the constitutional laws and the statute laws (and in reality the American legal system has many other levels than just those two).

Multiple Systems of Law

I am intrigued by the fact that we have so many different implementations of the Rule of Law. Every nation on Earth that abides by the Rule of Law has its own system of law. The ways in which the laws of nations interact is as varied as the relationships between the nations. For example, American Law has specific sections dealing with the fact that there are Native American "domestic dependent nations" within its borders that have their own laws.

Similarly, every nation has a different set of social conventions, those unwritten rules that lubricate our everyday interactions.

On top of all those different systems of law, we have International Law, with the intent of providing structure for interactions between nations when those nations have different and possibly incompatible systems of laws. Two aspects of International Law that I find particularly thought-provoking are the Law of War, and Jurisdiction.

(For an interesting bit of history about Jurisdiction, read about Peine forte et dure.)

Meta Law

In order to be predictable, the laws must be stable and not change often; but the laws must sometimes be changed in order to cover new situations or to correct problems in existing laws. One approach to improving the predictability of the system of laws while still allowing for change is to use a layered approach, where some laws are considered more important than others and are thus harder to change. The set of harder-to-change laws typically includes the rules on how to change the laws. This is the basis of the constitutional model, as is used in the United States, in which the most important laws are embodied in the constitution, with rules that make those laws much harder to change than regular laws. A constitution will typically include rules on how both "normal" rules and the rules embodied in the constitution can be changed.

Back in 1982, a "constitution" game by Peter Suber called Nomic appeared in Douglas R. Hofstadter's column, "Metamagical Themas," in Scientific American. In this game, players take turns proposing changes to the rules of the game. The rules start out in two categories, "immutable" and "mutable", corresponding to the simple two-level "constitutional" and "statute" law that Americans are taught in civics classes. The rules of the game tell how a player wins the game, and also tell how the rules can be changed - including how to change the rules that tell how to win and how to change the rules. The Nomic game is intended to illustrate the mechanisms and possibilities described in Peter Suber's book The Paradox of Self-Amendment, available online. For the quickest read on the game, you can jump straight to the rules, but the game description, although somewhat lengthy, is also interesting.

In Suber's book he starts by asking how a legal system can deal with paradox, when there are laws that directly contradict each other, and he notes that "paradoxes come and go without much notice and are dealt with without much ado."

Given that systems of laws seem always to be self-referential (since they include rules about how to change the rules), attempting to craft a system of laws that is also complete and consistent would seem to run into a version of Gödel's Incompleteness Theorem. In practice, systems of laws are not really complete and still blithely violate consistency, yet manage to be quite useful despite their flaws.

Law and Software

The title of this section might refer to laws that affect software, such as copyright law, or it might refer to the use of software to assist in the application of law, such as computerized law indexes or Regulation by Software; but in fact, I am referring to the use of law as a concept in defining how software works.

As in a society, a programming language is built on a set of rules that describe how statements in the language are interpreted by the computer. The developer uses his knowledge of these rules to create a program that instructs the computer to do something that is useful to the developer.

Imagine trying to program in a computer language with no rules. How could you get anything done? You could never predict the results of a statement, so you could never make a program that produced anything predictable.

Just as different societies each have their own set of rules, different programming languages each have their own set of rules. And just as with social conventions, different groups of programmers typically adopt programming conventions that are not enforced by the compiler but are intended to make life a bit simpler for the developers in the group.

In fact, all of the concepts discussed above are applicable to software. You can consider that as we take a look at what it means to define software in terms of laws.

Law and Object-Oriented Programming

Back in 1987 Naftaly Minsky and David Rozenshtein published "A Law-Based Approach to Object-Oriented Programming" (available for purchase on-line) in which they discussed how an object-oriented system can be described in terms of the laws that control the exchange of messages between objects. (Minsky has published quite a few other papers on related topics concerning law and software.)

They start by defining objects as containing state and program, with four primitive messages (prefixed by the octothorpe character, #) to create (#new) and destroy (#kill) objects and to get (#get) and set (#mutate) state. Messages are defined as a triplet of sender, message text, and target. Message delivery goes through the law system, which can take one of three actions:
  1. The message can be delivered to its target.
  2. The message text and/or target can be modified and then delivered.
  3. The message can be blocked and thus not delivered.
With these definitions and a permissive law that allows any object to send any message to any other object, the system does not exhibit many of the characteristics typically associated with object-oriented systems.

They then examine the effect of different kinds of laws, such as allowing primitive messages to be sent only by the same object. Through this approach they show how to implement common object-oriented features such as encapsulation, inheritance, and class variables as well as less common features such as multiple inheritance, exclusion of methods from inheritance, and triggers.

Given that the program is part of the state of the object, it can be modified with a #mutate message, so it is possible to describe self-modifying programs within this framework. The laws of the system control whether and how this message is allowed to be sent.

By defining the laws in objects that are themselves part of the system, those laws can then be changed. The system could start with a separate subset of laws that control how the laws can be changed, making this approach look very much like Suber's Constitution game.

The law system allows the laws to modify the content of a message or redirect it to a different target, allowing for the implementation of security checks and other forms of enforced delegation.

I am not aware of a production system that directly uses this style of law-based control of message passing, but there are some systems that use a conceptually similar method of applying a set of rules to some messages to control their delivery. For example, in the Java security model different environments can have different implementations of the SecurityManager, each with its own definition of the security policy (i.e. rules) that controls whether certain actions are allowed to be taken, which can be viewed as allowing the messages requesting those actions to be delivered. The OSGi security model goes further towards being a general law-based system, including the ability to specify rules via a string and to compose multiple security policies.

Law and Mind

For both societies and software, laws are rules telling us what we must and must not do, or do differently, and conventions are rules telling us what we should and should not do, or do differently. By following these rules and conventions, a society or a software system can be far more productive than one with the same underlying capabilities but where the rules and laws are less cohesive and effective or are not followed.

Could it be that the same is true for our minds? According to Marvin Minsky's theory of the mind as set forth in Society of Mind, our minds are composed of many small agents communicating with each other. Minsky's agents are very small pieces, and the communication between them is below our level of awareness. Perhaps our minds use something like Naftaly Minsky's law-based message delivery mechanism to monitor and control these low-level communications between agents.

Maybe the biggest difference between people who are productive and those who are not is in the different internal rules the two minds follow, and not so much a difference in raw underlying capability. Maybe productive people have a better set of mental rules controlling the messages within their minds. And if that is true, that leads to an interesting question: to what extent is it possible for people to rewrite their own low-level internal communication rules to improve their performance, and how might that be accomplished?

Sunday, July 6, 2014

From Obvious To Agile

What do you do when obvious isn't?

Installing new fence posts

Many years ago I had a fence that needed to be repaired. I got a recommendation for a fence repair man from a friend and had him come out to take a look. He said the panels between the posts were fine and did not need to be replaced, I just needed new posts. He quoted me a price for installing new fence posts that seemed quite reasonable, and I accepted his bid.

A few days later he came back to do the job. After he had been out there working for a while, I went out to take a look. I was surprised when I saw how he had installed the new fence posts. He had not removed the old posts and put new posts in their places, as I had assumed; instead, he simply planted a new post next to each old post and strapped them together. I was flabbergasted, and complained to him that my expectation was that he was going to take out the old posts and replace them with new posts. He was nonplussed. "I told you I would install new posts," he said. "Taking out the old posts would be way more work, and I would have to charge you more."

Well, he had me: he had indeed said only that he would install new posts. I was the one who assumed he would take out the old posts. I grumbled, paid him extra to replace a few of the old posts where it was particularly troublesome to have an extra post sticking out, and had the whole fence replaced the right way a few years later.

Keep using gmail

One of the startups at which I worked used gmail and was acquired by a large company that used Exchange. Concerned about the possibility of having to move to what we felt was a worse system, we asked what would happen with email. We were relieved when they said we could keep using gmail.

On the very first day that we were officially part of the new company, we were all told that we now had Exchange email accounts. "Hey!," we said, "you told us we could keep our gmail accounts." "Yes, you can," came the response, "but you also need to have an Exchange account for all official company email."

This was, of course, not what we had expected when we asked if we could keep our gmail accounts. But, as with the new fence posts, they had in fact kept their word and let us keep our gmail accounts; it was we who assumed that that would continue to be our only email account.

Everything under SCCS

At one of the places I worked, we hired a contractor to work on a subsystem. At one point we became concerned about how he was managing his source code, so we asked how he was doing that. "Everything is under sccs," he said. (This was well before the days of git, subversion, cvs, or even rcs; at the time, sccs (Source Code Control System) was what most people in our industry were using.) When he finally delivered the source code to us, we were annoyed to discover that he simply had a directory named "sccs", and all of his source code was contained in that directory; there was in fact no versioning or history.

Once again, this was not what we had expected. When he said "sccs" we assumed he was talking about the source code control system, when in fact he was just referring to a directory name; and when he said "under" we assumed he meant "managed by", when in fact he just meant "contained in."

A new and improved version of Android

My first smart phone was an Android phone running version 2.2. I watched as the newer versions of Android came out, filled with interesting new features. Finally, an over-the-air update was available for my phone. I eagerly updated and started playing with the new features. My first disappointment was with the new and definitely not improved performance: my phone was slow and laggy, and it no longer lasted even one day on a full charge.

I was even more dismayed to discover that they had removed USB Mass Storage Mode (MSC or UMS) and replaced it with a significantly less functional alternative, MTP (Media Transfer Protocol). In my case, it was completely non-functional for my use, because my home desktop machine was running Linux, and at the time there was not a working Linux driver for MTP mode.

I was, as you might expect, pretty ticked off. I had assumed without thinking about it that they would not remove a significant feature from a new version of the software, but they never said that.

Alternate Interpretations

Ask yourself: when reading the above anecdotes, did you realize in advance of the denouement what the problem would be for all of them? If it had been you, would you have made the same assumptions as I did?

Sometimes something seems so obvious to us that it does not even cross our minds that there might be an alternate interpretation.

I don't think it is possible for us to see these alternative interpretations in every case; often it is something with which we have had no experience, so could not be expected to know. We do, of course, sometimes consider alternative interpretations. In the future, if someone tells me they will install new fence posts, I will be sure to ask for more details. But we have to make assumptions as we deal with the world every day. If we examined every statement and every experience for alternative interpretations, that would consume all of our time, and we would not have any time left to pursue new thoughts. We learn to make instant and unconscious judgment calls: as long as what we hear and see has a high enough probability of an unambiguous interpretation, the possibility that there is an alternate interpretation does not bubble up to our conscious minds. Overall this is a very effective strategy that lets us focus our mental energies on situations where an unusual outcome is more likely. But this does mean that every once in a while we will miss something, with undesired results.

Going beyond obvious

I have already given my recommendation to State The Obvious. However, as you can see from the above anecdotes, this is not always enough. But what else can we do?

If you consider the anecdotes above, you might notice that, in most of them, by the time I realized that I had made an incorrect assumption, the deed was done and I was stuck with an undesired result. But the fence post story was a little different: in that case, I checked up on the work before it was done. Because I discovered the problem while it was happening, I was able to ask for changes and get a result that was closer to what I wanted.

Software Development

Not all of my blog posts are about software development, but in this case the application is obvious. Well, it seems obvious to me, but just in case it is not obvious to everyone, I will follow my own advice and explain in detail.

In the traditional waterfall process, a complete and detailed specification of the desired system is created before doing any of the implementation work. Once that spec is done, the system is built to match it. But, as we have seen from the anecdotes above, even a very simple spec, such as "install new fence posts", might be interpreted in a bizarre way that still matches the letter of the specification. In this case, the result might be something that arguably matches what was specified, but is not what was wanted.

Based on my personal experience and anecdotes I have heard from others, I believe that it is very difficult to write a good spec for something new, and impossible to write a spec that can not be interpreted by somebody in some bizarre way that satisfies the spec but is not the desired result.

Given that we can't guarantee that we can write a spec that will not be misinterpreted, what is the alternative? I think the only alternative is to do what I did in the fence-post case: check up on the work and make corrections along the way. This is embodied in a couple of the value statements in The Agile Manifesto: "Customer collaboration over contract negotiation" and "Responding to change over following a plan".

If you are asking someone to create something that is very similar to things that have been created before, and through previous common experience there is already a shared vocabulary sufficient to describe how the desired result compares to those previous creations, then you can perhaps write a spec that will get you what you want. The closer the new thing is to those previously created things, the easier that will be. But in software development, where the goal is often specifically to create something novel, this is particularly difficult. In that situation, I think that creating and then relying solely on a detailed spec is less likely to result in a satisfactory outcome; I believe an agreement on direction and major points, followed by keeping a close eye on progress, paying particular attention when something is being done for the first time, is the key to good results.

Writing a Spec

I'm not saying don't write a spec. I'm saying you need to recognize that a spec won't take you all the way, and a poorly written spec can hinder your progress. Writing a spec is like looking at a map and planning your route: often necessary but seldom sufficient. You need to be prepared for construction closures, blocking accidents, or even additional interesting sights you might decide to see along the way. For any of these diversions, you will need to reexamine your route in the middle of the trip and select an alternative. For a short trip, you might not run into any such problems and thus not need to modify your route, but the longer the journey the more likely that at some point you will need or want to deviate from your original route.

If you are familiar with the roads and have a clear destination, you might be able to dispense with the initial route planning completely: just head in the right direction and follow the signs. Or if you are on a discovery road trip and don't have a specific destination, then heading out without a planned route is fine. In most cases, though, some level of advance route planning will save time. You just need to stay agile and be prepared to change your route along the way.

Sunday, November 3, 2013

Code Guidelines

A list of basic goals for creating code.

In our team project at work, we wanted to have a set of style guidelines to allow everyone to more easily and quickly read the codebase and to avoid spurious code reformatting changes. As you might expect, there were different opinions on many points. To avoid fruitless "my way is just better" discussions, I wanted to step back and make sure we could all agree on some general goals. With that agreement in place, we could at least ask people to explain how their preferred style on some point supports our general goals. If nobody can provide an argument to support a favored construct, we might as well flip a coin.

Below are the goals I proposed and with which the team agreed. I think many of these are obvious, but then I usually believe in stating the obvious. The first two criteria below are also listed in my post on Software Quality Dimensions. Your team may choose slightly different guiding principles, but I think having the team agree on and write down their principles and asking people to justify their proposed standards against those principles can help short-circuit disagreements that might otherwise take longer to resolve.

Goals

In order of priority, with the most important criteria first: First, we want our code to be correct.
This means that the code must:
  • perform the desired primary behavior.
  • behave in a defined way for expected error conditions.
  • not have undesirable side-effects.
  • not have security vulnerabilities such as buffer overflows or injections.
  • not have memory problems such as leaks or use of released or uninitialized memory.
  • run fast enough for the intended use cases (but without premature optimization).
Second, we want our code to be robust.
This means that the code should be written in such a way as to minimize the probability of incorrect behavior under a wide range of conditions, including when:
  • it receives unexpected, corrupted, or no input data (graceful degradation).
  • a programmer unfamiliar with the code makes changes to it.
  • the functionality of neighboring code changes.
  • the development environment or toolset changes.
Third, we want our developers to be as productive as possible.
This means the code should be written such that:
  • developers are unlikely to misunderstand what the code does (principle of least surprise).
  • developers can read and understand the code quickly.

Wednesday, October 24, 2012

Role-Based Authorization

A simple, uniform, powerful and extensible authorization model.

Introduction

The "three As" of security are:
  • Authentication - assuring that the user is who he says he is.
  • Authorization - allowing each authenticated user to perform selected privileged actions.
  • Audit - recording privileged actions to allow review of changes or potential abuse of privileges.
Given authentication and auditing it is pretty simple to add a bit more monitoring that is very useful for billing purposes and resource management, so you more often see the combination AAA (Authentication, Authorization, Accounting) or AAAA (Authentication, Authorization, Audit, Accounting).

In this post I discuss only authorization. Authentication and auditing are each big topics, so I won't try to cover them here. Similarly, I assume that the code and data are themselves secure. In particular, I do not cover the issue of multiple security domains and the problem of having lower security code make requests to higher security code.

With my focus only on authorization, in the discussion below I assume that the user has been authenticated so that we can trust that piece of data within the application.

I will use the language of relational databases in this post because it is well-known and precise. An implementation of this model can use some other mechanism to store and query the authorization data. The SQL examples provide precision to the discussion, but you should be able to skip the SQL code and still gain a basic understanding of the model.

In the SQL example code I indicate replacement variables within braces; for example the string {user} in a SQL statement indicates that the application should plug in the user name at that point in the expression. For a real implementation, the actual syntax would depend on the database access package in use.

I have run into some authorization systems intended to provide a powerful set of capabilities for a complex situation that were, unfortunately, themselves so complex as to make it difficult to understand how they were supposed to work, and even after having it explained, difficult to remember because there was not a simple underlying model to tie it all together.

In this post I present an approach to authorization that I believe provides a very high level of power with a model that is relatively simple to understand and to extend as needed. This model initially implements a Role-Based Access Control (RBAC) mechanism, a widely used approach to security that is now a NIST standard. I add a few extensions to the common model that make it start to look more like an Attribute-Based Access Control (ABAC) model.

Separation of Concerns

In an authorization system, we want to separate the management of authorization from the application. The application should ask permission for what it wants to do, which permission is supplied by the authorization system. All management of the granting of the authorizations is handled from the authorization system, completely outside of the application. If you build a system in which any of the abstractions used in the management of authorizations, such as roles, appear in the application, then, as they say, you are doing it wrong.

In this post I focus only on the part of the system that determines whether to grant authorization. A separate system is required to maintain the data that is used by the authorization system. That maintenance can become quite complex in enterprise systems, but I will not be discussing it further in this post except to mention that the authorization mechanism described here can be applied to the system that maintains the authorization data in order to control who is allowed to modify what parts of that data.

Users

Let's start with perhaps the simplest useful authorization model possible. We begin with a one-column user table containing user names.

create table user(name varchar(32) primary key);
When the application wants to check for our sole authorization, it takes a passed-in authenticated user name and calls the authorization function with that value. The authorization function just checks to see if that user exists in the table. If so, the user is authorized and the authorization function returns true; if not, the user is not authorized and the authorization function returns false.
-- authorized if count>0
select count(*) from user where name={user};
The user-only model is too simple for most applications.

Actions

The next step is to add a one-column action table containing actions. We will assume each action is represented by a string name, although for performance reasons some might choose a different representation.
create table action(name varchar(32) primary key);
We add one row to this table for each restricted action; for example, we might have entries for login, reboot_system, and view_system_users.

With the addition of the action table we can no longer just look up users in the user table. We add a third table called grant (or auth_grant, since grant is typically a reserved word in SQL) with two columns that are foreign-key columns to the user and action tables. Each row of the grant table refers to a user and an action, with the meaning that that user is granted authorization to perform that action.
create table auth_grant(
    user varchar(32) not null,
    action varchar(32) not null,
    constraint FK_grant_user foreign key(user)
        references user(name),
    constraint FK_grant_action foreign key(action)
        references action(name)
);
Our authorization function will now accept a combination of values. We will refer to this combination as the requested operation (the NIST standard uses transaction as the unit for which permissions are granted). When an application wants to perform a potentially restricted operation, it takes the passed-in authenticated user name, adds the action it wants to perform, and passes that data to the authorization function. The authorization function takes the passed-in user and action arguments and looks in the grant table for a row in which the passed-in values for user and action match the values in the corresponding columns in the table. That row defines a permission to execute the requested operation. If that row exists, the operation is authorized; if that row does not exist, the operation is not authorized.
-- authorized if count>0
select count(*) from auth_grant where
    user={user} and
    action={action};
The user+action model is sufficient for many simple systems, such as granting login rights to some users and admin rights to other users.

Objects

With just users and actions, each action granted to a user effectively has global scope within the system. This is fine for actions such as login which truly are intended to be global in scope, but we would also like to be able to specify that certain actions can be performed on specific objects. Modern operating systems include mechanisms to grant different access rights, such as read-file or write-file, to specific files based on the user.

We add a one-column object table containing references to the objects in our system for which we want to be able to issue grants, with one row for each such object. We are making the simplifying assumption that each object already has a unique identifier that can be stored in our database.
create table object(name varchar(32) primary key);
We add a third column to our grant table that is a foreign-key column to the object table, exactly analogous to the existing references to the user and action tables. Each row of the grant table now refers to a user, an action and an object, with the meaning that that user is granted authorization to perform that action on that object.
create table auth_grant(
    user varchar(32) not null,
    action varchar(32) not null,
    object varchar(32) not null,
    constraint FK_grant_user foreign key(user)
        references user(name),
    constraint FK_grant_action foreign key(action)
        references action(name),
    constraint FK_grant_object foreign key(object)
        references object(name)
);
If we still want to have actions with global scope, such as the example of a login action in the user+action model, we can add a special system object that can be used in that situation.

Our authorization requests from the application now include three pieces of data. We modify our function for authorizing a restricted operation to take an argument specifying the object, along with the user and action arguments that we already have. The authorization function looks in the grant table as before, but it now must find a row that matches all three fields rather than only user and action.
-- authorized if count>0
select count(*) from auth_grant where
    user={user} and
    action={action} and
    object={object};
The user+action+object model presented here is used in many databases, with the objects being database tables or views and the actions being the four database actions of select, insert, update and delete. There may also be additional actions such as grant (the ability to create additional grants on an object) or actions that allow creating and modifying users or databases.

Roles

In order to simplify the maintenance of grants when we have a large number of users, we add a mechanism that allows us to group users together and grant permissions to a group of users rather than just to a single user. Users are grouped according to the roles they play; example roles are user, administrator, and superuser.

We add a role table with one row for each role we define. (We will look at other possible implementations later, but this choice serves well for explaining the concepts.)
create table role(name varchar(32) primary key);
In order to indicate which users have been granted (assigned) which roles, we add a user_role table with two columns: the user column is a foreign key to the user table that references the user, and the role column is a foreign key to the role table. A user having a role is indicated by adding a row to the user_role table referencing that user and that role. When granting authorization, a user will receive authorization for all roles he has.
create table user_role(
    user varchar(32) not null,
    role varchar(32) not null,
    constraint FK_userrole_user foreign key(user)
        references user(name),
    constraint FK_userrole_role foreign key(role)
        references role(name)
);
We also add a role column to our grant table. This column is a foreign key to the one column in our role table. A row in the grant table can now refer either to a user or to a role. It must reference one or the other; while it might be possible to set up a structure to enforce that constraint directly in the database, we will skip that exercise and instead suggest that this constraint could be enforced by an application-level database consistency check.
create table auth_grant(
    user varchar(32),
    role varchar(32),
    action varchar(32) not null,
    object varchar(32) not null,
    constraint FK_grant_user foreign key(user)
        references user(name),
    constraint FK_grant_role foreign key(role)
        references role(name),
    constraint FK_grant_action foreign key(action)
        references action(name),
    constraint FK_grant_object foreign key(object)
        references object(name)
);
The addition of roles is entirely an abstraction within the authorization system; the application is not aware of roles. An operation is defined by the same three values as before, and the application calls the authorization function in the same way as before to see if an operation is authorized, but the authorization function has to do a little more work now.

The application still passes the user, action and object arguments to the authorization function, and the authorization function still looks in the grant table to see if that combination of user, action and object is authorized, but now in addition to looking for a row that exactly matches those three values, it also looks up all of the roles the specified user has, and it looks for a row in the grant table in which the action and object values exactly match the values passed in and in which the role in the grant table is one of the roles the user has. If the authorization function finds a row that exactly matches the action and object and that exactly matches either the user or any of the user's roles then the action is authorized; if no such matching row is found then the action is not authorized.
-- authorized if count>0
select count(*) from auth_grant where
    (user={user} or role in (select role from user_role where user={user})) and
    action={action} and
    object={object};
The (user+role)+action+object model presented here has been used in the Unix filesystem for many years, with the objects being files and directories, the actions being read, write and execute/search, and the roles called groups.

In the NIST RBAC model permissions can only be assigned to roles, not to users. A strict implementation of this aspect could easily be implemented by dropping the user check in our authorization test (which also means we can drop the user column in the grant table):
-- authorized if count>0
select count(*) from auth_grant where
    role in (select role from user_role where user={user}) and
    action={action} and
    object={object};
Alternatively, we could think of each user as automatically being assigned a unique role whose name is the same as the user name. Or, we can choose never to assign any permissions to a user, only assigning them to roles.

Role Activation

The NIST RBAC standard includes a concept called Role Activation (or Role Authorization). When a user logs in, some subset of his roles can be activated. Allowing a user to activate and deactivate his assigned roles gives the user a way to ensure that he (or some program he is running) does not perform a privileged operation when he is not expecting it. Permissions are only granted for active roles, so even if a user has been given permissions through a role, a program will not be able to take advantage of them unless the user has activated a role that grants those permissions.

We can implement role activation globally by adding an is_active column to the user_role table.
create table user_role(
    user varchar(32) not null,
    role varchar(32) not null,
    is_active boolean not null default false,
    constraint FK_userrole_user foreign key(user)
        references user(name),
    constraint FK_userrole_role foreign key(role)
        references role(name)
);
When checking for authorization, we only include roles that are active for that user. If we continue to allow user-based permissions, then we would need to add an is_active flag for those permissions as well. When using activation it is simpler to exclude user-based permissions, as is done in the NIST RBAC model.
-- authorized if count>0
select count(*) from auth_grant where
    role in (select role from user_role where user={user} and is_active) and
    action={action} and
    object={object};
The NIST RBAC standard uses session-based activation rather than global activation. This allows a user to have multiple sessions open simultaneously with different roles active for each session. To implement this, rather than adding an is_active column to the user_role table, we create a session table that keeps track of our sessions and a session_role table that lists the roles that are active for each session.
create table session(
    id varchar(32) primary key,
    user varchar(32) not null,
    constraint FK_session_user foreign key(user)
        references user(name)
);

create table session_role(
    session_id varchar(32) not null,
    role varchar(32) not null,
    constraint FK_sessionrole_sessionid foreign key(session_id)
        references session(id),
    constraint FK_sessionrole_role foreign key(role)
        references role(name)
);
When testing for authorization we only want to use roles that are both assigned (in the user_role table) and active (in the session_role table). Assuming the mechanism that maintains active roles in the session_role table ensures that the only roles appearing in that table are in the user_role table (i.e. only an assigned role from the user_role table can be active in the session_role table), then we can modify the authorization function to accept an additional argument which is the session_id, and change our implementation SQL:
-- authorized if count>0
select count(*) from auth_grant where
    role in (select role from session_role where user={user} and session_id={session_id}) and
    action={action} and
    object={object};
At this point our model includes the capabilities of RBAC0, the first level of the NIST RBAC standard (although the NIST model does not include action and object as presented above). However, in order to keep the discussion of the other aspects of the model less cluttered, I will generally not be including role activation in the remainder of this discussion except where noted.

Role Hierarchies

Given the ability to group users into roles and thus simplify the number of grants we need to create, we can generalize on that concept by also allowing roles to be grouped into other roles.

In the discussion of Roles above, we added a user_role table that allowed us to assign roles to users. We now add a role_hierarchy table with parent and child columns that allows us to assign roles (children) to other roles (parents).
create table role_hierarchy(
    parent varchar(32),
    child varchar(32),
    constraint FK_rolehierarchy_parent foreign key(parent)
        references role(name),
    constraint FK_rolehierarchy_child foreign key(child)
        references role(name)
);
When collecting the list of roles for a user, we now have to recursively consult the role_hierarchy table to collect all of the child roles for any role the user has. How this is actually done is heavily dependent on the implementation. Some SQL databases include the ability to formulate recursive queries, but most do not.

We hide this implementation detail inside a view that collects the closure of the role-role relationships, effectively flattening our hierarchy. Defining this flattening in a view allows us to change how we collect the closure of the roles without affecting the queries that invoke this view. In this particular example, our view is defined using a non-recursive query that will suffice for a hierarchy of limited depth.
-- not a full closure if the hierarchy is too deep
create view role_closure as
    select distinct user, a3.child as role from user_role
        join role_hierarchy as a1 on user_role.role=a1.parent or
            (user_role.user=a1.parent and user_role.role=a1.child)
        join role_hierarchy as a2 on a1.child=a2.parent or
            (user_role.user=a2.parent and user_role.role=a2.child)
        join role_hierarchy as a3 on a2.child=a3.parent or
            (user_role.user=a3.parent and user_role.role=a3.child)
    ;
We can now use the role_closure view in place of the user_role table:
-- authorized if count>0
select count(*) from auth_grant where
    (user={user} or role in (select role from role_closure where user={user})) and
    action={action} and
    object={object};
If we want to use session-based activation, we can do that by modifying our role_closure view to be based on the session_role table rather than the user_role table:
-- not a full closure if the hierarchy is too deep
create view role_closure as
    select distinct session_id, user, a3.child as role from session_role
        join role_hierarchy as a1 on session_role.role=a1.parent or
            (session_role.user=a1.parent and session_role.role=a1.child)
        join role_hierarchy as a2 on a1.child=a2.parent or
            (session_role.user=a2.parent and session_role.role=a2.child)
        join role_hierarchy as a3 on a2.child=a3.parent or
            (session_role.user=a3.parent and session_role.role=a3.child)
    ;
As above when adding session-based role activation, the authorization SQL includes the session-id and we no longer allow user-based permissions:
-- authorized if count>0
select count(*) from auth_grant where
    role in (select role from role_closure where user={user} and session_id={session_id}) and
    action={action} and
    object={object};
This change can be made in any of the authorization SQL statements given below to add session-based authorization where it is otherwise not included.

Note that although we stated that the role parent/child relationships form a hierarchy, there is actually no reason to limit it to that, and our design does not preclude defining role relationships that form a more complex graph. We do want to avoid cycles in our role graph, as a graph with cycles would not provide us any useful benefits, and we need to ensure that our implementation does not blow up if the role graph happens to have some cycles. If we use the role_closure view implementation provided above, an incidental benefit is that the closure mechanism is so simple and limited, cycles will not cause any problems other than wasting a bit of processing power.

The NIST RBAC standard defines both general and restricted forms of hierarchy as part of the RBAC1 level. The restricted form is a tree structure and the general form is an arbitrary partial order. Our model above support the general form.

NIST RBAC levels RBAC2 and RBAC3 add Constraints (to ensure support of Separation of Duties) and Symmetry (the ability to review permission-role assignments as well as user-role assignments). With the simple database implementation presented here, these capabilities are available.

Alternate Hierarchy Implementations

In the implementation of user roles and role hierarchies above we added a role table, a user_role table and a role_hierarchy table, we added a role column to the grant table, we added a role_closure view and we modified our example SQL select statement for checking authorization to use that view. In this section I present three alternate approaches to this step when using a relational database, and of course there are other approaches not discussed here that are not based on a relational database. These implementation alternatives do not affect the basic model being developed.

In the first alternate approach, after defining the role table we next define the user_or_role view that is the union of those two tables.
create view user_or_role as
    (select name from user)
    union all
    (select name from role)
;
In the grant table, rather than adding a role column and having the user column be a foreign key to the user table, we make the user column a foreign key to the user_or_role view. Unfortunately, it is typically not possible to declare a foreign key to a view, in which case this foreign key relationship would have to remain implicit and not enforced by the database (it could be part of our application-level database consistency checks). Nonetheless, the SQL statements that join using this foreign key will work the same as if the foreign key were declared, although performance may be an issue if the user_or_role view can not be indexed. By using a materialized view it might be possible to index the view and have a foreign key refer to it, but then we would need to deal with rematerializing the view every time we changed the contents of the user or role tables.

Instead of creating a role_hierarchy table, we do the same thing to the user column of the user_role table as we did to the grant table, making it a foreign key to the user_or_role view rather than to the user table. This allows the user_role table to represent which roles have other roles as well as which roles users have directly been given.

In our second alternate implementation, we start by defining user_or_role as a table that contains the records for both users and roles, with an is_role column that indicates whether a row represents a user or a role. We then create user and role as appropriate views into that table.
create table user_or_role (
    name varchar(32) primary key,
    is_role boolean not null default false
);

create view user as
    select name from user_or_role where not is_role;

create view role as
    select name from user_or_role where is_role;
As in our first alternate implementation, the grant table points to the user_or_role table, as does the user column in the user_role table.
create table auth_grant(
    user_or_role varchar(32) not null,
    action varchar(32) not null,
    object varchar(32) not null,
    constraint FK_grant_user foreign key(user_or_role)
        references user_or_role(name),
    constraint FK_grant_action foreign key(action)
        references action(name),
    constraint FK_grant_object foreign key(object)
        references object(name)
);

create table user_role(
    user varchar(32) not null,
    role varchar(32) not null,
    constraint FK_userrole_user foreign key(user)
        references user_or_role(name),
    constraint FK_userrole_role foreign key(role)
        references role(name)
);
Many databases, including MySQL, do not allow indexes or foreign keys on views, so neither of the above two alternate implementations will work very well on those databases, and the table statements would have to be modified not to declare foreign keys to view columns.

If we want to use indexes and foreign keys, we have to compromise our data model a bit and not use views when we need foreign keys, which leads us to our final alternative.

In our third alternate implementation, we don't have a separate role table or view. Instead, we use the user_or_role approach as in the second alternative above: we place the role names into the user table and add an is_role column that indicates whether a row represents a user or a role.
create table user (
    name varchar(32) primary key,
    is_role boolean not null default false
);
In our user_role table, in which the role column was a foreign key to the role table, we make that column instead be a foreign key to the user table, where we are now storing our role names.
create table user_role(
    user varchar(32) not null,
    role varchar(32) not null,
    constraint FK_userrole_user foreign key(user)
        references user(name),
    constraint FK_userrole_role foreign key(role)
        references user(name)
);
We don't need a role_hierarchy table because we can now represent those role-to-role relationships in the user_role table. In our role_closure view we replace the role_hierarchy references with user_role references.
create view role_closure as
    select distinct a0.user, a3.role from user_role as a0
        join user_role as a1 on a0.role=a1.user or
            (a0.user=a1.user and a0.role=a1.role)
        join user_role as a2 on a1.role=a2.user or
            (a0.user=a2.user and a0.role=a2.role)
        join user_role as a3 on a2.role=a3.user or
            (a0.user=a3.user and a0.role=a3.role)
    ;
Because we are now storing our roles in the user table, the user column in our grant table can refer to either a user or a role, depending on what we are storing in the user table, so we don't need the role column and we can go back to the previous definition that did not have that column.

With this implementation our foreign key constraints all work because we are not dealing with any views, and our table structure is simpler because we have combined users and roles into one table. Although we are putting roles into the user table, we do need to remember that this is just a convenient fiction to simplify our implementation because there are some situations in which we want to treat users and roles the same. But we must remember that, although we are storing them in the same table and in some situations ignoring the difference between them, if we forget about that difference and start treating them the same in other situations we can easily start getting absurd behavior from our system.

(I have a mental image of our legal system as having a people table, and a law table with a foreign key to the people table. At some early point, someone wanted some laws that applied to corporations as well as people, so they said, "I know, let's just add an is_corporation flag to the people table and put the corporations in there, then our foreign keys from the law table will still work and we won't need to add a bunch more structure to our law schema!" With the passage of time, law programmers who should have been paying attention to the is_corporation flag started ignoring it more and more often, until finally the law programmers were saying, "Well, those corporations are in the people table, so they must be people." If you are concerned that this kind of situation might happen to you, you might not want to put roles into the user table.)

For the remainder of this discussion, we will use this third alternate implementation approach.

Interlude

In the above discussions, I have been assuming that the names of users, actions, objects and roles are also their key values. This implies that each of those names are unique. Given that I have discussed a couple of implementations in which users and roles have been mixed together, you might wonder whether it would cause problems to add a user whose name is the same as a role. In the above simple implementation the answer is "yes", and the system would have to disallow that. A real system is likely to be a bit more complex, using unique IDs as primary keys rather than names. The problem of having unique names thus gets moved from a database issue to an application-level issue. The system implementer must decide under what circumstances it is acceptable to have duplicate names, and there must be a way to distinguish those duplicates to someone operating the system.

We have reached a point in the development of our authorization model that is similar in power to many existing systems. People who need more flexibility than this model provides might diverge at this point into custom authorization systems with various forms of exceptions and extensions that rapidly start adding complexity to the model.

There are still a number of extensions we can make to our authorization model that will improves its power while adding only a small amount to the cognitive load of understanding how it all works. Let's get back to our model and add some more power to it.

Tasks

In the same way that we allow specifying a group of users having a role, we add the ability to specify a group of actions, which we call a task. The relation between tasks and actions is exactly analogous to the relation between users and roles. Each action can be assigned to multiple tasks, a task can be assigned other tasks, and an authorization grant can refer either to an action or to a task.

Analogous with our second alternative implementation above, in which we added an is_role column to the user table and put roles into the user table, for the equivalent addition of tasks we add an is_task column to the action table, add an action_task table with columns action and task both being foreign key references to the action table, and add a task_closure view.
create table action(
    name varchar(32) primary key,
    is_task boolean not null
);

create table action_task(
    action varchar(32) not null,
    task varchar(32) not null,
    constraint FK_actiontask_action foreign key(action)
        references action(name),
    constraint FK_actiontask_task foreign key(task)
        references action(name)
);

create view task_closure as
    select action, a3.task as task from action_task as a0
        join action_task as a1 on a0.task=a1.action or
            (a0.action=a1.action and a0.task=a1.task)
        join action_task as a2 on a1.task=a2.action or
            (a0.action=a2.action and a0.task=a2.task)
        join action_task as a3 on a2.task=a3.action or
            (a0.action=a3.action and a0.task=a3.task)
    ;
We expand our authorization query to look for tasks in the same way as we expanded it to handle roles, with the same caveats about hierarchy depth.
-- authorized if count>0
select count(*) from auth_grant where
    (user={user} or user in (select role from role_closure where user={user})) and
    (action={action} or action in (select task from task_closure where action={action})) and
    object={object};

Domains

Roles and tasks give us the ability to group users and actions. We complete the pattern by adding the ability to group objects into groups that we call domains (not to be confused with internet domain names). As with the tasks example above, we add the is_domain column to the object table, create the object_domain table to allow defining groups of objects, create the domain_closure view, and modify the authorization function to check for either objects or domains in the same way as we modified it to check for either actions or tasks. All of these steps are exactly analogous to what we did when we added tasks.

Intermediate Summary

Let's take stock of what our model looks like:
  • There are three dimensions: user, action, and object.
  • The handling of the three dimensions is completely symmetric (unless role activation is being used, in which case the user dimension has that extra wrinkle).
  • The application passes those three values to the authorization function, which returns true if that operation is authorized, false if not.
  • For each dimension, there is a grouping mechanism: role for user group, task for action group, domain for object group.
  • The grouping mechanism supports a hierarchy of groups, or more generally a (directed acyclic) graph of groups (a partial ordering).
  • To determine if a request should be authorized, take each dimension, collect the closure of the groups for that dimension, and look for a grant in which each dimension of the grant matches any of the items in the closure for that dimension.
The model presented above is easy to understand, but despite its simplicity it is quite powerful. Yet it does not suffice for everyone. Let's see how we can continue to enhance it's power without significantly increasing its complexity.

Times, Periods and Schedules

In some systems it is desirable to allow some operations only at specified times. For example, one might want to allow users to log in to the system only during their work shift.

We define another dimension, the time dimension, and we define a time range as a period, where a period is an interval of time such as 8AM to 5PM, or Sunday, or 8AM to 5PM on weekdays. We add the time dimension to our definition of an operation, so when the application calls the authorization function, it must now pass the current time as a fourth argument.

The dimensions we have defined previously are all discrete dimensions, with only one matching value for each definition. The time dimension is different in that it is a continuous dimension: there are multiple time values that can match a period. This makes the authorization function a little more difficult to write, but it does not add much complexity to the user's conceptual model.

The other dimensions all have groups, so it would not add to the complexity of the model to add groups of periods. In fact, the model would be more complex if we did not add groups of periods, as that would make this dimension different from all the others in that aspect, which would be an additional detail that the user would have to factor into his mental model.

We add a group called schedule. As with all the other groups, a period can be included in any number of schedules, and schedules can contain other schedules. When checking authorization, we collect all the periods that match the current time and the closure of all the schedules for those periods, and we search for grants that include any of those in the period column.

Locations, Areas and Regions

By now the pattern should be pretty clear. If the system requires other dimensions, they are easy to add by following the same pattern. By keeping to the pattern, the complexity of the model that the user must work with to understand the system is kept low, even when there is some small difference for the new dimension, as there was for the time dimension when compared to the three previously defined dimensions. When there are small model extensions for a dimension, as there was when we added the time dimension, we can leverage that model concept when adding some other dimension.

Location is a system-specific concept. For some systems it might be a logical location, such as "console", "secure terminal", or "dial up". Since these are discrete values, it would suffice to have a location table, group locations in regions, and handle it in the same manner as the other discrete dimensions such as user.

For other systems a location might mean a physical location specified by one or more continuous values, such as latitude and longitude, in which case we define an area analogously to a period, where one area includes a range of locations. The area might be defined with a center point and radius, it might be defined with a bounding box, it might be defined as a polygon, using splines, or in some other even more complex way. As with periods, the complexity of the definition of an area has an effect on the difficulty of implementing the authorization function that has to determine whether a location is or is not in an area, but has little effect on the complexity of the user's mental model of the authorization. For the user, it is sufficient to know that a given location will be either contained in or not contained in an area, and that grants are based on areas.

Our group for an area is a region, and it groups together areas and other regions in the same way as the groups in the other dimensions.

Denials

The approach described above is essentially a "whitelist" approach, which is the standard approach to authorization. If an operation is listed in the grant table then it is allowed; any operation which is not listed is not allowed.

It is also possible to use a "blacklist" approach: rather than allowing what is listed and denying everything else, we can deny what is listed and allow everything else. In this case we would create a denial table that is exactly like the grant table except that it contains operations to be denied rather than operations to be allowed. The authorization function would do the same search as before, except that it would deny the operation if any matching records were found, and allow the operation otherwise.

Using a blacklist approach to authorization as just described is generally not recommended (in fact the NIST RBAC standard specifically recommends against "Negative permissions", although it does not outright disallow them). Since the default action is to allow an operation, if a new operation is added to the system and through oversight the appropriate denials are not added, then there is no protection for the new operations.

Exceptions

We can combine the original grant approach and the denial approach described just above to give us the ability to have both a whitelist and a blacklist. We start with our original grant table approach, following the recommended position that the default is to deny any operation unless it is explicitly granted; on top of that, we add the denial table as exceptions to the grants.

Our authorization function first looks in the denial table; if a matching record is found, then the request is denied. If no matching record is found, then the function looks in the grant table; if a matching record is found, then the request is granted; otherwise it is denied.

This allows the admin to think in terms of exceptions: grant privileges to all of X, except for Y. In some situations this allows expressing the intended grants more simply than if one is restricted to just additive grants.

We could also flip the grant and denial tables around, first looking in the grant table for a match, then looking in the denial table for a match, then granting if nothing is found. As discussed in the previous section, this is not recommended, but understanding that it is possible is conceptually useful, and leads us to our last enhancement.

Prioritization

The structure of the grant and denial tables are identical, and their contents are checked in the same way, with the only difference being an inversion of the interpretation of the results in one case as compared to the other. We can easily combine both of these tables into a single auth table that includes an additional allow column that is true for all records from the grant table and false for all the records from the denial table. We can also add a priority column that we use to determine which records we should attend to first.
create table auth(
    id integer auto_increment,
    allow boolean not null default true,
    priority integer not null default 0, -- higher values take precedence
    user varchar(32) not null,
    action varchar(32) not null,
    object varchar(32) not null,
    period varchar(32) not null,
    area varchar(32) not null,
    constraint FK_auth_user foreign key(user)
        references user(name),
    constraint FK_auth_action foreign key(action)
        references action(name),
    constraint FK_auth_object foreign key(object)
        references object(name),
    constraint FK_auth_period foreign key(period)
        references period(name),
    constraint FK_auth_area foreign key(area)
        references area(name)
);
If we define the priority value such that higher values are more important than lower values, then we can get the same behavior as described in the first part of the previous section by setting the priority on all the denial records to 2 and setting the priority on all the grant records to 1. Our authorization function then looks in the auth table for the matching record with the highest priority value and looks at the allow value for that record.

If we wanted to get the (non-recommended) behavior as described at the end of the previous section, we could do that by setting the priority of all the grant records to 2 and setting the priority of all the denial records to 1, plus making the default behavior (when no matching rows are found) to allow the operation.

Given this structure, we can of course put in records with any priority value. This allows building up a series of toggling exceptions, much as the way leap years in the Gregorian calendar are defined (each year has 365 days, except every 4th year is a leap year with 366 days, except every 100 years is not a leap year, except every 400 years is a leap year).

Since we can stack up alternating grant and denial records, the only distinction between the "whitelist" and "blacklist" approaches discussed earlier is the question of what the default is when no matching records are found in the auth table (the default for whitelisting is deny, the default for blacklisting is grant). Given that using a default of allow is not recommended, we define the system to use a default of deny, but we provide a way that the system can effectively be set up with a default of allow if desired.

To simulate a default of allow, the admin can create a group for each of the dimensions in our authorization model (user, action, etc) that includes all elements of that dimension. Thus there would be an AllUsers role, an AllActions task, an AllObjects domain, etc. The admin then creates a rule that includes all of these groups with allow set to true and priority set to zero. Since the rule has been defined to include all elements of every dimension, it will always match every operation, so there will never be a case where there are no matches and the system default of deny is used. Assuming all other priority values are greater than zero, this rule will be the lowest priority, so it will only have an effect if there are no other matches, and thus it acts as the default.

As described above, there is one more potential ambiguity to resolve: what happens if there are two rules with the same priority but opposite allow values? (Two rules with the same priority and the same allow are not a problem, as they both give the same result.) We resolve this ambiguity by defining the denial records to take precedence over the grant records when they have the same priority value. This definition reduces nicely to the desired behavior for the simplest denial+grant case when all records have the same priority.

Our authorization function thus looks for all matching records in the auth table, sorts first by priority then by allow, picks the first one, and uses its allow value to determine whether to allow the operation. If no matching records are found, the operation is not allowed.

Ignoring for now the more complicated portions of the WHERE clause for selecting time and location, here is our SQL statement for determining if an operation is authorized:
-- The single selected value is true if authorized; if false or no records, not authorized
select allow from auth_grant where
    (user={user} or user in (select role from role_closure where user={user})) and
    (action={action} or action in (select task from task_closure where action={action})) and
    (object={object} or object in (select domain from domain_closure where object={object}))
    order by priority desc, allow asc
    limit 1
Adding prioritization like this adds a new concept to the authorization model, but provides a good amount of additional power relative to the additional mental load to understand the model. However, creating well-structured rules using prioritization is trickier that it seems at first glance. It has the same essential problem as for the blacklist approach described above: mistakes in setting up the conceptual layers of the different levels of prioritization can result in unexpected security holes. If you can figure out how to set up your authorizations using grants only, without denials, you should do that. But if the grant-only model is not sufficient, then adding prioritization as described in this section is a reasonable way to take the model to the next level of power - just remember that you have to be more careful in how you set up your rules.

Summary

With the addition of prioritization in the previous section, our authorization model is complete. Let's review the complete model.
  • There are two kinds of dimensions: discrete and continuous.
  • There are five dimensions: user, action, object, time and location.
  • User, action and object are discrete; time is continuous; location can be either discrete or continuous, depending on how the system defines it.
  • Additional dimensions can be added if necessary, following the pattern of the existing dimensions.
  • The handling of every discrete dimension is completely symmetrical with every other discrete dimension (unless session-based role activation is included, in which case the user dimension is a little different); the handling of each continuous dimension is close to completely symmetrical with the other continuous dimensions; and there is a high level of symmetry between the discrete and the continuous dimensions.
  • The application passes a value for each dimension to the authorization function. This collection of dimension values is the operation for which the application is requesting authorization. The authorization function returns true if that operation is authorized, false if not.
  • For each continuous dimensions, there is a range defined as the basic match: period for time, area for location.
  • For each dimension, there is a grouping mechanism: role for user group, task for action group, domain for object group, schedule for period group, region for area or location group.
  • The grouping mechanism supports a hierarchy of groups, or more generally a (directed acyclic) graph of groups.
  • There is a set of rules that is used to determine whether an operation is authorized. Each rule includes a set of comparison values, one for each dimension, a priority, and an allow flag that tells whether that rule specifies that authorization for a matching operation should be granted or denied.
  • To determine if a request is authorized, take the value for each dimension in the request, collect the closure of the groups for that value, and collect the records in which each dimension of the grant matches any of the items in the closure for that dimension. Pick the record with the highest priority, giving preference to deny records over grant records, and use the allow value of that record to determine whether to authorize or deny the operation. If no matching records are found, the operation is denied.
This conceptual model is no longer trivial, but the above rules are still relatively concise and easy to understand. The model is general enough and powerful enough that it should be suitable for a wide variety of applications.

In our model the application passes in a set of values to the authorization function, which uses its abstractions (in the form of groups) and rules (in the form of prioritization) to determine whether or not to grant permission for an operation. If we need more power, the application can pass in additional information, whether it is additional attribute information about the user, the environment, or other aspects of the operation, and the authorization system can apply even more complex rules. This is the approach used by Attribute-Based Access Control, with a rules engine used in place of the mechanisms described here.